vLLM 0.32.0 adds structured-decision API reading first-token probabilities
AccBalanced · x · 2026-10-19
vLLM 0.32.0 introduces a structured-decision API (/v1/systemone): given a fixed set of legal answers, it reads the model's first-token probabilities instead of generating JSON token by token. Supports choice, yes/no, and score modes with ordinary generative models (demoed with Qwen3-0.6B). With prefix caching, multiple questions over the same state prefill once — each extra decision is question → one-token read → probabilities. The release (814 commits) also adds NVFP4/FB8 KV cache and GPU n-gram speculator Model Runner V2.
More from coding & agent
- Greptile raises $50M Series B at $750M valuation, reviews 1% of all GitHub PRs — Rasmic · 2026-10-19
- Do you still review every UI change by hand when an agent writes it? — mfbzme · 2026-10-19
- Dev builds playable Tesla Roadster web game in a weekend with Opus 5.5 — edwinarbus · 2026-10-19
- Blender renders cut from 4 hours to minutes with Opus 5.5 and cloud GPUs — ravisparikh · 2026-10-19
- LangChain Hosts Life Sciences Agent Engineering Workshop: Build, Trace, Evaluate Research Agents — LangChain · 2026-10-19
- Librarian Agents: new paper lets agents curate their own long-context workspaces — charles_irl · 2026-10-19