vLLM serves DeepSeek-V4.1-Flash at 954K tokens/s per GPU, matching DeepInfer at 7.9× throughput
Xianbao_QIAN · x · 2026-10-25
The vLLM team details how to serve DeepSeek-V4.1-Flash at scale on a single NVIDIA GB300 NVL72 rack.
- With P/D disaggregation and Wide-EP, vLLM hits 954K tokens/s per GPU on SemiAnalysis AgentX (P90 of 196 tokens/s per user) — a 7.9× jump over its earlier single-node result
- Cost works out to about $38.96 per GPU-hour at DeepSeek's off-peak prices
- A specialized decode path reaches 826 tokens/s for a single user
- vs. DeepInfer, DeepSeek's own engine: decode throughput is within 6% across batch sizes 8–256, and prefill tracks DeepInfer's published curve — showing a general-purpose engine can match a model-specific one
The post breaks down each optimization: understand the model first, small DEP instances for prefill, Wide-EP scaling with EPLB, and CPU overhead becoming critical at NVL72 scale. All built on existing vLLM components and DeepSeek's open-sourced kernels.
Related event: vLLM matches DeepSeek's DeepInfer, hits 954K tokens/s per GPU on GB300(9 posts)→
More from Infra
- gpu-unlock repo claims to unlock hidden tensor core speed on RTX 3090 Ti: 87 to 157 TFLOPS — jfischoff · 2026-10-25
- DeepInfer is deeply tailored to DeepSeek 4.1F: KV cache of just 890 bytes per token — teortaxesTex · 2026-10-25
- Reasoning costs plummet as startup Talaas matches System 1 speeds — where's the alpha? — Square-Listen-7989 · 2026-10-25
- NextDC plunges 40% in a day: the real AI bottleneck is the power line, not contracts — ccerrato147 · 2026-10-25
- A neocloud is a mortgage on a data center: rent comes later, interest is due now — ccerrato147 · 2026-10-25
- Korean AI data center firm Elice falls 32% on debut despite 400x oversubscription — ccerrato147 · 2026-10-25