DeepSeek open-sources DeepInfer: inside V4.1 Flash's 155T tokens/day inference system
teortaxesTex · x · 2026-10-20
DeepSeek published an engineering deep dive on Zhihu unveiling DeepInfer, the open-source inference framework behind its large-scale V4.1 Flash service, supporting both NVIDIA and Huawei Ascend hardware.
Key production numbers
- 155 trillion tokens/day serving throughput from a single cluster
- 96% input cache hit rate; 1.5M tps per-GPU prefill throughput with cache hits
- 400ms average TTFT (P99 1000ms)
- 230 tokens/s generation, up to 300 tokens/s for coding
- KVCache retained 72h after inactivity; P:D ratio near 1:1, $38/GPU-hour revenue at off-peak pricing
Architecture: vision encoding, prefill and decode run as separate stages scaled independently — prefill on standard 8-accelerator servers, decode on 64–128-accelerator supernodes with wide expert parallelism; stages exchange state via external KVCache storage and dedicated load balancers. The post also includes prefill throughput-vs-latency curves (EP4DP4, 384+32 experts, 131K context) showing throughput saturating at larger batch sizes.
Related event: DeepSeek Open-Sources DeepInfer Inference Framework(24 posts)→
More from Infra
- Chrome 155 ships JPEG XL: 30-50% smaller than JPEG with lossless JPEG transcode — banteg · 2026-10-20
- Polymarket: DeepSeek releases open-source tool to sharply cut AI inference compute costs — Polymarket · 2026-10-20
- Together, Fireworks, Prime Intellect and Modal are all converging on inference + RL training — NielsRogge · 2026-10-20
- NVIDIA GTC kicks off in Berlin with Jensen Huang keynote set for tomorrow — NVIDIAAI · 2026-10-20
- Hyperdrives CEO interview: cheaper power-dense electric motors via cooling breakthrough — MarvinTBaumann · 2026-10-20
- MiniMax H3 video generation runs on a single RTX 3060 12GB via GGUF quantization — Ok-Inevitable8391 · 2026-10-20