DeepSeek has officially open-sourced DeepInfer, the high-performance inference framework that powers its production services for DeepSeek-V4.1-Flash. This release, published under the Apache 2.0 license, includes the companion deepseek-recipe library, which handles token encoding and decoding to match the official API behavior. The announcement hit Hacker News with over 2,100 points within hours, marking the first time a frontier lab has released its entire production serving engine rather than just isolated components.

Under the Hood

DeepInfer is not a simple wrapper; it is a deeply modified fork of vLLM, incorporating inspiration from SGLang and TensorRT-LLM. The framework bundles optimized kernels such as FlashMLA, DeepGEMM, DeepEP, and TileKernels. It supports advanced features like mixture-of-experts parallelism, asynchronous scheduling, and deterministic sampling. Crucially, it includes DSpark speculative decoding, which uses a small draft model to propose five tokens at a time, verified by a confidence head to reduce latency and boost throughput by 4 to 7 percent in production loads.

Production Performance Metrics

The performance numbers are staggering for a self-hosted solution. A single DeepSeek cluster processes 155 trillion tokens per day, achieving off-peak revenue of approximately $38 per GPU per hour. By leveraging a prefix cache that hits 96 percent of inputs, per-GPU prefill throughput jumps from 60,000 to 1.5 million tokens per second. First-token latency sits at around 400 milliseconds, with generation speeds reaching 230 tokens per second. The SWA Bounded Replay technique further optimizes costs by fixing cache-miss recomputation to the last 128 tokens of a prompt, nearly halving total prefill compute for long contexts.

Hardware and Limitations

DeepInfer supports both NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs, requiring Linux, Python 3.12, GCC 13 or later, and either CUDA 13 or CANN 9.2. This dual support is significant for Chinese firms aiming to reduce reliance on NVIDIA while serving global customers. However, the release is not a complete serving stack; KV cache management and cross-machine load balancing remain closed due to tight coupling with DeepSeek's internal infrastructure. Currently, only offline inference for DeepSeek-V4.1-Flash is supported, though future model inference code will land in the same repository.

Key Takeaways

  • DeepInfer is a full production-grade engine, not just a demo or isolated kernel library.
  • Performance metrics show a 25-fold increase in prefill throughput via aggressive prefix caching.
  • Dual hardware support for NVIDIA Blackwell and Huawei Ascend 950 signals a strategic shift in Chinese AI infrastructure.
  • KV cache management and load balancing remain proprietary, limiting full self-hosted parity.

The Bottom Line

This is a massive win for builders who want to escape the pricing pressure of Western cloud providers. If you have the hardware, DeepInfer gives you a practical path to dramatically lower serving costs, though you still need to solve the load balancing puzzle yourself.