Structured-decision trick speeds up DiffusionGemma inference 3-10x with one forward per request
bodonoghue85 · x · 2026-09-17
Developer mmastrac sped up Google's DiffusionGemma 3-10x by borrowing Jev's ideas; the implementation is merged into his diffgemma repo (PR #21).
How it works: diffusion models already operate on probabilities—by pinning the structured tokens of the response in place, most denoising work can be skipped. A JSON question schema in the system message turns a /v1/chat/completions request into a single scored denoise forward: the answer template is seeded into the canvas with label slots as noise, and label distributions are read from logits at each slot. Every question costs one forward and each slot conditions bidirectionally on the whole template, modeled after TypeSafe's Jev API (choice/score/yes-no with probabilities and confidence). The author suggests the approach could be ported into vLLM.
More from Infra
- Dev proposes predictive dynamic context caching for Claude Code — Sauers_ · 2026-09-17
- Starlink expands across Latin America: 10,000 antennas to connect 8,000 schools in Honduras alone — NicoVerderosa · 2026-09-17
- Massive inference run on Fable cost $1,050 per second, dev reveals — Sauers_ · 2026-09-17
- Massive inference run costs $1,050 per second, researcher shares sticker shock — Sauers_ · 2026-09-17
- Nebius raises on-demand GPU/CPU prices for the second time this year — kevinsxu · 2026-09-17
- RL training costs cross $1M, offering a rare look behind the scenes — zainhas · 2026-09-17