vLLM v0.32.0 lands: 814 commits, low-precision KV everywhere, up to 85% higher req/s

vllm_project · x · 2026-10-18

vLLM v0.32.0 is out with 814 commits from 363 contributors (121 first-timers).

Highlights

• New /v1/systemone structured-decision endpoint; generic hf tool/reasoning parsers read the checkpoint's responsetemplate, removing model-specific parsers
• Low-precision KV cache everywhere: NVFP4 on SM8x/90/12x; GLM-5.x defaults to FP8 KV on SM100, halving KV memory with 2.3–5.5% more throughput at a 7–11% TTFT cost; ROCm gets 4-bit KV with 2x FP8 density
• Model Runner V2: GPU n-gram speculator, Elastic EP, PP+CP for MLA, default on ROCm
• Faster startup & serving: precompiled kernels, autotune preload, +43–85% req/s with multi-API workers
• New model support: EmbeddingGemma 2, MiniCPM-V 4.7, Nemotron 3.5 ASR; 3 security fixes + CORS hardening

Original post →

More from Infra

• Cadence announces sovereign AI cloud for European chip designers — karlfreund · 2026-10-23
• SGLang Deep-Dive Blog Series Breaks Down KV Cache, Posts up to 15.5x Throughput Gains — BanghuaZ · 2026-10-23
• DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
• Capital Is Now the AI Buildout Bottleneck, Says Analyst Citing Jensen Huang — JOBhakdi · 2026-10-23
• Cerebras explains how it deploys trillion-parameter models on wafer-scale chips — bookwormengr · 2026-10-23
• IFM open-sources RL360, the agentic RL infra behind K2 Horizon's training — rupspace · 2026-10-23

Infra channel →