vLLM upgrade guide: KV offloading, queue admission control, 33.6% Blackwell latency cut
vllm_project · x · 2026-09-11
The vLLM project published a pre-upgrade rundown of serving, frontend, and performance changes:
- Serving/frontend: Mooncake Store can now offload decode KV; new queue admission control flags (--max-num-queued-reqs, --max-num-queued-tokens); a /v1/messages/render endpoint for the Anthropic Messages API; Rust frontend adds audio/video inputs over gRPC; bounded cachesalt length fixes a scheduler CPU-exhaustion DoS.
- Performance: Kimi-K3 Mamba metadata prep fused into a single Triton launch (6.6–7.6x at kernel level); per-architecture batch invariance yields 3x decode kernels on RTX 4090D / H20; Blackwell autotuning cuts end-to-end latency 33.6%.
- Hardware: NVIDIA reports DeepSeek V3.2 and GLM-5.2 DSA reaching CUDA implementations on every GPU; AMD ROCm defaults W4A4 to the preshuffled asm GEMM.
Related event: vLLM v0.29.0 Cuts Blackwell Latency 33.6%, Model Runner V2 Default(4 posts)→
More from Infra
- Taiwanese families at TSMC's Arizona plant are shocked US high school is 'so much easier' — Polymarket · 2026-10-27
- OpenRouter hits 1 trillion tokens per day — ycombinator · 2026-10-27
- Engineer reverse-engineers Canon printer firmware to kill ink counters and proprietary cartridges — ngxson · 2026-10-27
- AI Cloud Operators Sell Years of Compute Yet Struggle to Finance Delivery — AccBalanced · 2026-10-27
- Modal engineer's deep dive into how vLLM and SGLang inference engines work — charles_irl · 2026-10-27
- TSMC reportedly preparing another CoWoS price hike of up to 20% as AI packaging demand outstrips supply — rohanpaul_ai · 2026-10-27