DeepSeek open-sources DeepInfer: cluster serves 155T tokens/day at $38 per GPU-hour
dl_weekly · x · 2026-10-26
DeepSeek has open-sourced DeepInfer, the inference framework behind its production services, targeting the core problem of serving massive user bases at reasonable cost.
- One cluster serves 155 trillion tokens per day, revenue at off-peak pricing works out to $38 per GPU per hour
- Currently supports only DeepSeek-V4.1-Flash; runs on Nvidia Blackwell (SM100) and Huawei Ascend 950, with no Hopper support
- Speculative decoding via DSpark: a draft model proposes 5 tokens at a time, a confidence head predicts acceptance, cutting checked draft tokens 30% per step and boosting throughput 4–7% at production load
- A second technique reruns only the last 128 tokens of each prompt through decoder layers, nearly halving total prefill compute
- Bit-identical output features and all published benchmarks are Nvidia-only and self-reported
More from Infra
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- xAI's Colossus 1 hits ~14.6 exaflops FP64, ~5x Top500 leader — cis_female · 2026-10-27
- Lambda runs 19 B200 nodes in a 16-node power budget, boosting throughput 24% — TheZachMueller · 2026-10-27
- Podcast: Micron's HBM roadmap, Meta's AI infra, and NVIDIA's supply chain lock-in — BenBajarin · 2026-10-27
- SGLang MegaKernel + PD Pushes the TCO-Latency Frontier on GB300, SemiAnalysis Benchmarks Show — AccBalanced · 2026-10-27
- Scaling self-hosted Convex: compiling TypeScript Convex functions to WASM — leventov · 2026-10-27