DiffusionGemma-Jev lands in vLLM: single-step structured answers with confidence scores
bodonoghue85 · x · 2026-09-23
vLLM has merged a structured generation mode for DiffusionGemma-Jev (PR #57250, driven by mmastrac), targeting yes/no, multiple-choice, and scored questions where every answer comes with a confidence score.
The technique: vLLM seeds a canvas with the response template, keeps only the answer slots noisy, then reads a probability distribution from every slot in a single denoising step — no autoregressive decoding.
Google's Gemma team paired the release with a one-command Cloud Run deployment: 35-60ms single-step latency, 100-123 req/sec at batch@32, roughly $3/hr with scale-to-zero when idle, so no GPU of your own is needed.
More from Infra
- Four Mac Studios Over Thunderbolt Run Trillion-Parameter Model on One Wall Outlet as Apple Pitches Local AI — mark_k · 2026-09-23
- Archgen Labs, aiming to make chip design 1,000x faster, lands YC backing — retr0jirachi · 2026-09-23
- Running Omarchy and a 90M-parameter LLM on a PSP — Kyrannio · 2026-09-23
- A Redditor built a dual AMD R9700 local inference rig and crowdsources tuning advice — Current-Ticket4214 · 2026-09-23
- stuntd: a local proxy that trains a free model to replace paid LLM yes/no calls — Inevitable-Log5414 · 2026-09-23
- OpenRouter Batch API spans 71 models, auto-routes to cheapest provider — jeff_weinstein · 2026-09-23