DeepSeek open-sources DeepInfer, the inference framework powering its production services
jiqizhixin · x · 2026-10-20
DeepSeek has released DeepInfer, the high-performance inference framework behind its large-scale production services, under the Apache-2.0 license. The repo hit 1.1k stars shortly after launch.
The initial release supports DeepSeek-V4.1-Flash and ships the core features for offline serving at scale:
- Optimized kernels, model and sampler runtime
- Parallelism: Attention DP / balanced CP, MoE EP + EPLB
- DSpark and confidence-scheduled verification
- SWA bounded replay
- Asynchronous scheduling with a zero-overhead batch scheduler
- Batch-invariant inference and deterministic sampling
- Structured output
It includes dependency configs for both NVIDIA and Huawei Ascend hardware, plus full engineering directories (eval, examples, tests). This is the first open-source release of DeepSeek's internal production inference stack.
Related event: DeepSeek Open-Sources DeepInfer Inference Framework(21 posts)→
More from Infra
- MiniMax H3 video generation runs on a single RTX 3060 12GB via GGUF quantization — Ok-Inevitable8391 · 2026-10-20
- ByteDance, Alibaba, Tencent place over $35B in Nvidia chip orders amid supply crunch — pstAsiatech · 2026-10-20
- DeepSeek-V4.1-Flash tops Agent Arena open models, ranks #11 overall at $0.20/task — vista8 · 2026-10-20
- vLLM at 94,235 GitHub stars: the high-throughput LLM inference engine keeps climbing — vllm-project · 2026-10-20
- From-scratch GPT-2 124M pretrained on 2.48B tokens with 4x A40 — full writeup — upulbandara · 2026-10-20
- New Whale inference engine claims to crush vLLM and SGLang — EAccelerate_42 · 2026-10-20