DeepSeek open-sources DeepInfer inference stack: 155T tokens/day, 96% cache hit rate
赛博禅心 · wechat · 2026-10-20
DeepSeek open-sourced its DeepInfer inference framework (Apache 2.0) and deepseek-recipe codec component (MIT), the stack behind V4.1 Flash. Revealed production stats: 155T tokens processed daily per cluster, 77K average input tokens, 96% KV-cache hit rate, and $38 revenue per GPU-hour.
Key details:
- PD disaggregation: Prefill on 8-GPU servers, Decode on 64–128 GPU supernodes at 1:1 ratio; KV cache stored in a 100PiB all-flash 3FS cluster (3-day retention for agent traffic, 7-day for chat), boosting Prefill throughput 25x (60K → 1.5M tokens/s).
- SWA Bounded Replay: cache misses recompute only the last 128 tokens at fixed cost, cutting total Prefill compute nearly in half.
- DSpark speculative decoding with accept-probability filtering adds 6% throughput; GPU/CPU overlap scheduling. Decode runs at 14K tokens/s per GPU, 230 tokens/s per user (300 for code).
- MoE load balancing: 384 routers spread across 64 GPUs with hot-expert replicas, peak load only 9% above average.
The open-source version supports offline inference on NVIDIA SM100 and Huawei Ascend 950; cache storage, PD deployment and replica balancing remain internal. DeepInfer is built on vLLM, referencing SGLang and TensorRT-LLM.
Related event: DeepSeek Open-Sources DeepInfer Inference Framework(24 posts)→
More from Infra
- Raw data shows OpenAI returned 271 overload errors to a Codex pool in 48 hours — rickasaurus · 2026-10-20
- Researcher claims 4x faster convergence to backprop loss using Flywheel infra — tensorqt · 2026-10-20
- Chrome 155 ships JPEG XL: 30-50% smaller than JPEG with lossless JPEG transcode — banteg · 2026-10-20
- Polymarket: DeepSeek releases open-source tool to sharply cut AI inference compute costs — Polymarket · 2026-10-20
- Together, Fireworks, Prime Intellect and Modal are all converging on inference + RL training — NielsRogge · 2026-10-20
- NVIDIA GTC kicks off in Berlin with Jensen Huang keynote set for tomorrow — NVIDIAAI · 2026-10-20