vLLM hits 954K tokens/s per GPU serving DeepSeek V4.1-Flash on GB300 NVL72 rack
vllm_project · x · 2026-10-25
vLLM announces that with P/D disaggregation and Wide-EP on a single GB300 NVL72 rack, it serves DeepSeek V4.1-Flash on the SemiAnalysis AgentX benchmark at 954K tokens/s per GPU with 196 tokens/s P90 per user — 7.9× its single-node result. Credits DeepSeek's openness, close NVIDIA collaboration, and SemiAnalysis' AgentX; engineering led by the inferact team, with an accompanying engineering blog.
More from Infra
- AWS Interconnect goes multicloud: GA with Google Cloud, OCI added, Azure in preview — DavidLinthicum · 2026-10-25
- CMU professor jokes about funding his lab by reselling B300 compute as DeepSeek tokens — niloofar_mire · 2026-10-25
- Hyperscaler depreciation to hit $590B by 2029, likely exceeding combined operating profits — GaryMarcus · 2026-10-25
- vLLM matches DeepInfer on DeepSeek-V4.1-Flash across batch sizes 8–256 — vllm_project · 2026-10-25
- UK per-capita electricity now below 1970s levels, just as AI demands power — NinaDSchick · 2026-10-25
- Hyperscaler AI capex hits $800bn, 2.4% of US GDP, dwarfing Manhattan Project — Unfair-Journalist-66 · 2026-10-25