vLLM squeezes 5.3x throughput from DeepSeek-V4.1-Flash: 1.3B tokens per dollar
woosuk_k · x · 2026-10-17
The vLLM blog details three weeks of optimization after DeepSeek-V4.1-Flash's launch: a 1.9× speedup at low concurrency and 5.3× throughput under a 150 TPS constraint, hitting 1.3B tokens per dollar on SemiAnalysis AgentX — 830K tok/s per GPU (cached + input + output) at 100 tok/s per user.
Two optimization tracks:
- SWA bounded replay with CUDA graphs, cutting TTFT by 30%;
- Kernels: integrating DeepSeek's MegaAttention, Mega-mHC, Mega-Gate and DeepSelect, plus aggressive fusion/parallelization of remaining kernels (mHC side streams, fusing all-reduce with adjacent ops into one kernel).
Model-side architecture matters too: V4.1's causal encoder-decoder (CED) activates 16B params during decode but only 8B during prefill; CSA2 + FP4 KV cache + inter-layer KV sharing push global KV footprint to 890 bytes per token. Agentic throughput is up 5× since day 0.
More from Infra
- Running LLMs on a Commodore 64, a 486 DOS PC and a Nintendo Wii — Awacer · 2026-10-17
- Buy vs rent vs API for inference: it all hinges on your sustained GPU utilization — Dizzy_Lingonberry947 · 2026-10-17
- New prototype geoclass pools OpenAI subscriptions to bypass regional inference capacity limits — TheZachMueller · 2026-10-17
- How distributed LLM training actually works: 6 parallelism strategies, comms and memory bottlenecks — techNmak · 2026-10-17
- Punny take on data center fears: the clouds are at their limits — gnukeith · 2026-10-17
- HF engineer teases data center setup running 4 agents at 256k context, 100 TPS — TheZachMueller · 2026-10-17