DeepSeek open-sources DeepInfer: 155T tokens/day, $38/GPU-hour, >80% margin
Latent Space · rss · 2026-10-21
Latent Space dissects DeepSeek's open-sourcing of DeepInfer, the production inference stack behind V4.1 Flash, timed with PyTorch Conf.
Key numbers
- 155T tokens/day per cluster, 96% input cache hit, 400ms TTFT, 230 tok/s generation (300 for coding)
- At $5-8/card/hour all-in cost and $38 revenue per accelerator-hour, >80% inference margin; roughly $166B per GW with an 81-day payback
Open-source boundaries
- Built as a vLLM customization (Apache 2.0); supports NVIDIA and Huawei Ascend, speculative decoding, SWA Bounded Replay, structured output; 72-hour prompt caching vs OpenAI's 12
- But the crown jewels — KV-cache management (8+ hour lifetime), PD disaggregation, load balancing — remain unreleased, so "fully open source" ≠ reproducible production service
Architecture highlights
- Separate vision/prefill/decode stages; scheduling by estimated compute rather than token count; cache-affinity-free prefill via shared prefix storage; decode balancing request count plus KV length; expert replication with diminishing returns; async CPU scheduling
- No tensor parallelism in the disclosed Blackwell config, traced back to V3-era H800 interconnect constraints
Response
vLLM team acknowledged DeepInfer's origins and published matched-config GB200 results within 6% of DeepSeek's published decode throughput, countering claims that the efficiency is out of reach for open-source engines. Latent Space cautions these are unaudited first-party claims; margin conclusions are sensitive to cache accounting, hardware cost, and workload mix.
More from Infra
- antirez tries DHH's Omarchy on a Dell XPS, plans to adapt DwarfStar SSD model streaming — antirez · 2026-10-21
- NVIDIA reportedly sits on $35B in China orders but can supply only 5-10% — emmanuelvivier · 2026-10-21
- Nvidia's China orders reportedly hit $35B, but supply may cover just 5–10% — emmanuelvivier · 2026-10-21
- willcb: open-sourcing a frontier lab's prod stack is cool, but nothing majorly new today — willcb · 2026-10-21
- swyx: AI compute is supply-constrained — more tokens means more revenue, not lower prices — swyx · 2026-10-21
- Qwen3.8-Flash-Next runs stable at 1.66bpw exl3, fits entirely on one 5090 — Unstable_Llama · 2026-10-21