vLLM v0.34.0 Lands: 1,061 Commits, Up to 1.95x Faster Decode on GB200
vllm_project · x · 2026-11-19
vLLM v0.34.0 is out, bundling 1,061 commits from 412 contributors (120 new):
- New models: 10 additions including Mistral Large 4, MiniMax-M3.1-Flash and Nemotron-3.5-Transcribe.
- Extensible KV cache on by default: vLLM now sizes the KV cache from memory actually free after warmup, giving most models a bigger cache and more throughput; gpumemoryutilization now defaults to 1.0.
- Model Runner V2 is the only GPU runner: V1 is removed, and VLLMUSEV2MODELRUNNER=0 now fails at startup.
- Performance: big speedups on Blackwell and MI355X — up to 1.95x Step5 NVFP4 decode on GB200, 1.6x throughput on GLM-5.3-Flash, and 1.24–1.40x faster long-prompt TTFT on GLM-5.3.
Upgrade notes: default block size is now 64 (was 16; ROCm paged attention still 16), ROCm 7.2 images are gone, and huggingfacehub >= 2.0.0 is required. These changes can break existing setups; migration steps are documented.
More from Infra
- DNeo: Better Target Feature Encoding Speeds Up Speculative-Diffusion LLM Inference — cerebras · 2026-11-19
- Cascade MoE on Bittensor beats Salesforce's Moirai2 on GIFT-Eval 19 days after launch — const_reborn · 2026-11-19
- Sciforium hits 1,129 output tokens/sec P90 on DeepSeek-V4.1-Flash over 4 AMD MI355X GPUs — rohanpaul_ai · 2026-11-19
- Running untrusted agents on GKE: sandboxing, cold starts, and state — rseroter · 2026-11-19
- OpenAI models find up to 14% PPA gains on AMD GPU IP, improve custom Jalapeño chip — OpenAI · 2026-11-19
- A Local Model for Every Device: 6 Free Offline Models from Phones to Servers — alex_verem · 2026-11-19