vLLM serves DeepSeek-V4.1-Flash on Vera Rubin NVL72 at 58x GB300 throughput, 387 tokens/s per user
vllm_project · x · 2026-11-13
The vLLM team reports serving DeepSeek-V4.1-Flash on a single NVIDIA Vera Rubin NVL72 rack at 1.7M tokens/s per GPU while each user still gets 387 tokens/s — 58x GB300 NVL72 — with peak throughput of 2.52M tokens/s and a latency-optimized path hitting 1,810 tokens/s (P90) at batch size 1.
Key optimizations:
- Rubin-tuned kernels: reworked MoE, attention, GEMM, mHC and indexer kernels keeping bandwidth-bound data in each SM's memory domain and using SM107's wider MMA instructions; 28% faster DEP2 prefill, 25% faster decode.
- Specialized TP decode path with cross-operator fusion and compute/communication overlap, spread TP8 across two trays.
- Rack-scale memory pooling: a Mooncake KV pool across all tray DRAM and spare HBM, Engram tables read over NVLink, sustaining 96.5-97% hit rates, letting Dynamo route to the least-loaded rank (TTFT P90 0.98s → 0.38s).
Work led by Inferact and the vLLM team, with DeepSeek, NVIDIA, the Mooncake team and SemiAnalysis.
Related event: vLLM Optimizes Rubin: DeepSeek-V4.1-Flash Hits 58x GB300 Throughput(4 posts)→
More from Infra
- AWS disputes outage reports: services 'have been fine', check Health Dashboard — Kyrannio · 2026-11-13
- The math says GPU export controls are working: only a sliver of Nvidia output leaks — bookwormengr · 2026-11-13
- NIDO ships fully offline open-source Android AI assistant with encrypted local RAG and E2EE mesh — NIDO_007 · 2026-11-13
- NVIDIA researcher mocks US compute roadshow: 'Huawei and DeepSeek just do their own thing' — suchenzang · 2026-11-13
- Azure SQL Always Encrypted: running range queries on data the database can't read — adnan_hashmi · 2026-11-13
- Liquid AI's LFM hits 'college graduate' intelligence at 120 tps on a MacBook Pro — JosephJacks_ · 2026-11-13