vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU on GB300 NVL72 rack
woosuk_k · x · 2026-10-25
vLLM announced that with P/D disaggregation and Wide-EP on a GB300 NVL72 rack, it serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU with 196 tokens/s P90 per user — 7.9× its single-node result, detailed in a new engineering blog.
DeepSeek researcher Yifan Qiao argues it's easy to make inference benchmarks look impressive with random inputs and idealized expert routing; their benchmarks instead use real input distributions, realistic routing, and serving configs validated on AgentX. The chart shows GB300 NVL72 (Dynamo vLLM) dominating H100, MI355X, B200 and other configs on the throughput-vs-interactivity curve. Results have been online since Oct 20, before DeepInfer's release, with more speedups promised.
More from Infra
- Aggregated 18 ComfyUI GPU-unlock benchmarks show BF16 gains dwarf INT8 — Acceptable_Put9983 · 2026-10-25
- Blogger revisits Kimi K3 hype, speculates China may release some B300 GPUs — lipeng0820 · 2026-10-25
- H100/H200 GPUs are sold out this week as model releases and openkernelchallenge overload providers — abhijithneil · 2026-10-25
- Musk warns of 'stranded compute' in partially built data centers as GPU rental prices soar 2-3x — elonmusk · 2026-10-25
- Survival LLM: Free Offline AI Assistant Bundling 1.5M Emergency Articles, No Internet Needed — burny_tech · 2026-10-25
- SGLang vs vLLM: the inference framework race has no weekends — AccBalanced · 2026-10-25