DeepSeek open-sources DeepInfer: 155T tokens/day throughput on one cluster
智东西 · wechat · 2026-10-20
DeepSeek open-sourced DeepInfer, the internal large-scale inference framework behind V4.1 Flash, plus deepseek-recipe (production-grade tokenizer/codec), alongside a detailed Zhihu tech report.
Key numbers: one cluster serves 155T tokens/day at $38/GPU-hour (valley pricing). Throughput tricks include intra-node MoE expert parallelism, small EP for prefill vs 64–128-way Wide-EP for decode, four load balancers on one cost model, DSpark variable-length speculative verification (+6% throughput), and async scheduling. KV cache is retained 72h+ and block-cached in a separate 3FS cluster (100 PiB SSD, 10 TiB/s reads) so any prefill machine can hit any prefix; SWA stays local with 98% hit rate, and SWABoundedReplay cuts decode-side prefill compute nearly in half. Latency work covers staged-parallel Rust tokenization and streaming tool-argument JSON emission. Runs on NVIDIA and Huawei Ascend; KV management and load-balancing code remain internal.
Related event: DeepSeek Open-Sources DeepInfer Inference Framework(24 posts)→
More from Infra
- Chrome 155 ships JPEG XL: 30-50% smaller than JPEG with lossless JPEG transcode — banteg · 2026-10-20
- Polymarket: DeepSeek releases open-source tool to sharply cut AI inference compute costs — Polymarket · 2026-10-20
- Together, Fireworks, Prime Intellect and Modal are all converging on inference + RL training — NielsRogge · 2026-10-20
- NVIDIA GTC kicks off in Berlin with Jensen Huang keynote set for tomorrow — NVIDIAAI · 2026-10-20
- Hyperdrives CEO interview: cheaper power-dense electric motors via cooling breakthrough — MarvinTBaumann · 2026-10-20
- MiniMax H3 video generation runs on a single RTX 3060 12GB via GGUF quantization — Ok-Inevitable8391 · 2026-10-20