Early vLLM PR adds Jev-like structured generation for DiffusionGemma, only 2x endpoint latency on a DGX Spark
generativist · x · 2026-09-17
Developer mmastrac filed an early vLLM PR (#57250) adding a Jev-like structured generation mode for DiffusionGemma.
- Approach: fix canvas positions in a structured output format, have the model emit a token plus logprobs so the client derives entropy, and resample when entropy exceeds a threshold to reach agreement/confidence.
- Latency: reportedly only 2x the model's published endpoints on a single DGX Spark, which the author calls remarkable for an open model.
- Caveat: currently requires a structured-response server between vLLM and the client; folding it into the PR is planned.
More from Infra
- llama.cpp fails to load Qwen3.8 MTP draft model: 'output_hc_norm.weight' tensor not found — Ambitious_Fold_2874 · 2026-09-17
- Prediction: Kimi K3-level AI on a single RTX 5090 within 18 months — TheZachMueller · 2026-09-17
- User says he'd pay $1k/month for AI, but high pricing makes local AI attractive — draginol · 2026-09-17
- Tencent open-sources FlexKV distributed KV cache for LLM inference, cutting TTFT by up to 70% — Roger_M_Taylor · 2026-09-17
- NVIDIA releases NVFP4 quantized DeepSeek-V4.1-Flash on Hugging Face — TheZachMueller · 2026-09-17
- Optimization mined via Bittensor competition lands in vLLM, boosting Qwen3 throughput ~4% — const_reborn · 2026-09-17