Structured-decision trick speeds up DiffusionGemma inference 3-10x with one forward per request
bodonoghue85 · x · 2026-09-17
Developer mmastrac sped up Google's DiffusionGemma 3-10x by borrowing Jev's ideas; the implementation is merged into his diffgemma repo (PR #21).
How it works: diffusion models already operate on probabilities—by pinning the structured tokens of the response in place, most denoising work can be skipped. A JSON question schema in the system message turns a /v1/chat/completions request into a single scored denoise forward: the answer template is seeded into the canvas with label slots as noise, and label distributions are read from logits at each slot. Every question costs one forward and each slot conditions bidirectionally on the whole template, modeled after TypeSafe's Jev API (choice/score/yes-no with probabilities and confidence). The author suggests the approach could be ported into vLLM.
More from Infra
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02
- Researcher hoards nearly 1TB of DRAM, refuses to sell a stick to data centers — kevinnbass · 2026-11-02
- Self-Hosting LLMs at a Law Firm: Infra and Data Governance Harder Than the Model — EducationalGroup7245 · 2026-11-02