DeepInfer: DeepSeek's open inference framework
DeepSeek has released the inference stack behind its large-scale services, exposing specialized scheduling, parallelism, and decoding techniques, but the current release is model- and hardware-specific and does not include every component needed for a complete service.
Logged at IST: 2026-10-20
What it is: DeepSeek has open-sourced DeepInfer, the inference framework it says powers its large-scale production services. The repository currently supports DeepSeek-V4.1-Flash and describes the release as an offline-inference stack with core serving features.
Gist: The implementation combines optimized model/runtime kernels with attention and expert parallelism, asynchronous scheduling, multi-token draft verification, structured outputs, and options for batch-invariant inference and deterministic sampling. The README lists NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs as supported hardware, with platform-specific prerequisites.
DeepSeek publishes prefill and decode performance charts, measured on four SM100 GPUs for prefill and 64 for decode within one scale-up domain. Those are project-reported results, and the repository provides a sweep script for reproducing them. DeepInfer is Apache-2.0 and acknowledges that it builds primarily on vLLM, with inspiration from SGLang and TensorRT-LLM.
Caveat: This is not a turnkey, general-purpose serving platform yet. The repository says a KV-cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas are not included in this release. Its supported model and accelerator matrix is also narrow.
Source: https://github.com/deepseek-ai/DeepInfer Saved X post: https://x.com/deepseek_ai/status/2112476496283775135