DeepSeek open-sources DeepInfer, its inference engine

What shipped

DeepInfer is the inference framework behind DeepSeek's large-scale cloud service. DeepSeek says a single cluster serves 155 trillion tokens a day with revenue of $38 per GPU-hour at off-peak pricing. The open release currently supports DeepSeek-V4.1-Flash.

• Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels.
• Parallelism: attention DP and balanced CP, MoE expert parallelism with EPLB.
• Speculative decoding (DSpark) with draft tokens and confidence-scheduled verification.
• Batch-invariant inference and deterministic sampling, NVIDIA only for now.
• Structured outputs constrained by regex, JSON schema or EBNF grammar.

It ships alongside deepseek-recipe, which converts between API protocols and token sequences using the same logic as the official DeepSeek API, so a self-hosted deployment behaves like the original.

What is missing

The README lists what a complete online service still needs: a KV cache system, prefill/decode disaggregation and load balancing across replicas. These are tightly coupled to DeepSeek's internal infrastructure. Their design is described in the deployment Q&A and a Zhihu post. In short, this is a strong engine, not a finished service.

How much hardware

Per the Q&A, the smallest efficient unit is roughly 64 GPUs for prefill and 64 for decode, with EP64 to EP128 recommended for decode. The system is designed to keep KV cache for more than 72 hours, budgeting about 14.4 TB of logical storage per GPU on a 3FS storage cluster.

For smaller teams this is more a learning resource and kernel source than a stack to download and run. For anyone serving DeepSeek models at scale, it is the first official reference from the model maker. The Huawei Ascend support is telling too: DeepSeek openly optimizes for two hardware platforms.