Google's DiffusionGemma: Retrofitting LLMs into Diffusion Models at 10% of the Cost
The Decoder · rss · 2026-08-09
Google DeepMind introduced DiffusionGemma, demonstrating that text diffusion models can be built without training from scratch. By retrofitting the existing Gemma 4 model, researchers achieved this using less than 10% of the original training budget.
Unlike traditional autoregressive approaches that predict tokens sequentially, this diffusion-based model generates 256 tokens in parallel, reaching speeds of approximately 1,500 tokens per second. However, in benchmark evaluations, its overall generation quality—particularly in reasoning tasks—still trails behind the original autoregressive model.
More from Models
- willcb: Opus 5.5 doesn't smell bigger, it smells like they fixed the data — willcb · 2026-10-16
- Unverified claim: Apple and Meta's 816B-parameter Muse Spark Open runs on two Mac Studios — TansuYegen · 2026-10-16
- Gemini 4 model card: model eliminated months of trial-and-error on bioweapons research — Puzzleheaded-King584 · 2026-10-16
- Every AI picks 7: Anthropic engineer's rule for escaping model taste convergence — rubenhassid · 2026-10-16
- Brain teaser: strip all attention, MLP and embedding matrices from a Llama 3 — how many trainable params remain? — Sauers_ · 2026-10-16
- Lean v4.35.0 ships verified kernels — one written and proven consistent by Claude — geoffreyirving · 2026-10-16