vLLM hits 954K tokens/s/GPU on GB300 NVL72, matching DeepSeek's DeepInfer

vllm_project · x · 2026-10-25

vLLM's engineering blog 'Serving DeepSeek-V4.1-Flash at Scale' reports:

Matching DeepInfer: across batch sizes 8–256 on DeepSeek's decode configuration, vLLM's decode throughput is within 6% of DeepSeek's own engine, with prefill tracking DeepInfer's published curve — running DEP32 vs DeepInfer's EP64DP64. DeepInfer was originally built on vLLM.
Rack-scale serving: with P/D disaggregation and Wide-EP on a GB300 NVL72, vLLM serves SemiAnalysis AgentX at 954K tokens/s per GPU with 196 tokens/s P90 per user — 7.9× its single-node result — at $38.96/GPU-hour at DeepSeek's off-peak prices. A specialized decode path reaches 826 tokens/s for a single user.
Lessons: understand the model before optimizing; small DEP instances work best for prefill; Wide-EP decode scales well with EPLB; CPU overhead becomes critical at NVL72 scale.

Takeaway: a general-purpose engine, via deliberate engineering on existing components plus DeepSeek's open kernels, can match a model-specific one.

More from Infra