vLLM pools rack-scale KV cache on Rubin, holding 96.5-97% hit rate at high concurrency
vllm_project · x · 2026-11-13
The vLLM team details serving optimizations for NVIDIA's Vera Rubin: with far higher per-rack concurrency, live prefixes outgrow any per-rank cache, so they pooled the rack's memory via a Mooncake KV pool across DRAM and spare HBM, plus Engram tables sharded over NVLink — sustaining a 96.5-97% hit rate even at very high concurrency. Freed from rank pinning, Dynamo's least-loaded routing cut TTFT P90 from 0.98s to 0.38s. Rubin-tuned kernels also sped DEP2 prefill by 28% and decode by 25%.
Related event: vLLM Optimizes Rubin: DeepSeek-V4.1-Flash Hits 58x GB300 Throughput(4 posts)→
More from Infra
- "Singapore" reportedly accounts for 15-20% of NVIDIA revenue, and the jokes write themselves — JensHonack · 2026-11-13
- SGLang Summit agenda: LinkedIn serves ads model at 10K+ QPS under 80ms — DavidBennett__ · 2026-11-13
- Open weights leak which GPU trained them — six Chinese models read as NVIDIA Blackwell — Antique_Atmosphere97 · 2026-11-13
- Meme explains CUDA perfectly: optimized for many Alice Huangs, not one — djcows · 2026-11-13
- Perplexity's Mac app shows a local models option, hinting at offline capability — dnlkwk · 2026-11-13
- 2-bit DeepSeek V4.1 Flash runs a coding-agent seat on two 2016 V100s: a 12-hour real-work trial — a4mula · 2026-11-13