DeepSeek Opens Up DeepInfer: The Engine Serving 155 Trillion Tokens a Day, Minus Cache and Load Balancing

News Summary

DeepSeek has fully open-sourced DeepInfer, the in-house inference framework it uses to serve its models in production, together with deepseek-recipe, a companion library that converts text to tokens and back exactly as the company's official API does. The announcement was posted on X on Tuesday, October 20, 2026 (the Next Web timestamped its report at 2:08 pm UTC, roughly 10:08 am Eastern Time), and the code is published on GitHub under the Apache 2.0 license.

What DeepSeek Released

DeepInfer lives in the GitHub repository deepseek-ai/DeepInfer. DeepSeek introduced it on X with the words "Introducing DeepInfer, our in-house inference framework, now fully open source!" and described its purpose as "Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency."

An inference framework is the software layer that runs a trained model for real users. It decides how requests are batched, how work is spread across many chips, and how quickly each token is produced. SiliconANGLE, in a report updated at 7:11 pm Eastern Time on October 20, said DeepSeek calls the cost of serving large models to many users "the core problem an inference framework has to solve."

The repository bundles optimized code for DeepSeek's models, tools for distributing work across many accelerators, speculative decoding, features that keep outputs identical from run to run, and support for forcing answers into set formats such as structured outputs. DeepSeek also says inference code for its future models will be added to the same repository.

The deepseek-recipe Companion Library

deepseek-recipe was released alongside the framework. It converts incoming requests into token sequences using the same logic as DeepSeek's official API. The goal is consistency: teams who self-host DeepSeek models should see the same tokenization and request handling behavior that users of the hosted service get. One technical blog described the library as written in Rust with Python bindings, but that detail comes from a single source.

Supported Models and Hardware

For now, DeepInfer supports a single model, DeepSeek-V4.1-Flash, which launched in September. By default it loads the updated "1017" version released over the weekend of October 18.

Supported hardware includes Nvidia SM100 (Blackwell) data-center GPUs and Huawei Ascend 950 NPUs. Hopper and older Nvidia generations are not listed. Two of the reproducible-output features currently work only on Nvidia hardware, and all of DeepSeek's published benchmarks were run on Nvidia chips. Ascend support follows a September 30 release that brought the TileLang kernel language to Huawei chips.

Technical Highlights

SiliconANGLE detailed two efficiency techniques described in the release:

• Speculative decoding (DSpark): a small draft model proposes five tokens at a time, and a confidence head predicts which ones the main model is likely to accept. Verification compute is spent only on the likeliest candidates, which cuts the number of draft tokens checked per step by about 30%. DeepSeek estimates a 4% to 7% throughput gain at its production load.
• Reduced prefill: only the last 128 tokens of each prompt pass through the model's decoder layers, which DeepSeek says nearly halves total prefill computation.

A deployment Q&A published with the code says one cluster serves 155 trillion tokens per day, and that off-peak revenue works out to about $38 per GPU per hour. These are DeepSeek's own figures and have not been independently verified. Reports also name SWA Bounded Replay among the supported capabilities.

What Is Not Included

The release is not a complete serving stack. Key-value cache management and load-balancing code are not published, because DeepSeek says they are tightly coupled to its internal infrastructure; a Zhihu blog post describes how both work. Splitting prompt processing (prefill) and answer generation (decode) across separate machines is left to operators, although the README says most of the engine work needed for it is already in place.

The smallest deployment DeepSeek describes as efficient uses 64 GPUs for prefill and 64 for decoding. According to the Next Web, its benchmark setup used four SM100 GPUs for prompt processing and 64 for output generation. Bloomberg noted that no independent benchmarks exist yet.

Lineage and Community Response

The new README names vLLM as DeepInfer's original foundation, with input from SGLang and Nvidia's TensorRT-LLM. It also builds on kernel libraries DeepSeek released earlier: FlashMLA, DeepGEMM, and DeepEP. This is a shift from April 2025, when DeepSeek said its full inference engine was an early vLLM fork entangled with its cluster management tools and that maintaining a large open-source project was beyond the bandwidth of a small research team.

The response was quick. By Tuesday afternoon the repository had passed 8,000 GitHub stars, and the vLLM project publicly thanked DeepSeek and said it would apply ideas from DeepInfer to its own V4.1-Flash support. Some Chinese-language coverage gave different dates for the open-sourcing and described the release as an offline inference version, so details may be refined as the repository evolves.

Why It Matters for Developers

For researchers and engineers, DeepInfer offers a rare look at how a major model provider serves its models at scale, from speculative decoding to prefill savings. Cloud providers and self-hosting companies could adopt these methods to reduce serving costs. The Next Web also noted that open-weight models account for 23% of spending on Vercel's AI Gateway, a sign of how much demand there is for transparent, self-deployable AI infrastructure.

Sources: SiliconANGLE, The Next Web, and Chinese-language coverage including IT之家.