DiffusionGemma Report: Parallel 256-Token Generation Breaks AR Bottleneck

SungjinAhn_ · x · 2026-08-05

Google released the technical report for DiffusionGemma, an experimental open-weight discrete diffusion language model. Unlike conventional autoregressive (AR) models that decode sequentially, it iteratively refines blocks of 256 tokens in parallel, overcoming the sequential decoding bottleneck.

The model is obtained by fine-tuning the Gemma 4 Mixture-of-Experts model, which has 25.2B total and 3.8B activated parameters. The compute-efficient two-stage training pipeline includes:

Supervised Fine-Tuning (SFT): To teach bidirectional denoising.
Reinforcement Learning & Sampler Distillation: To jointly improve generation quality.

This pipeline uses fewer than 10% of the original AR model's training token budget. Additionally, the model integrates the core mechanism of Loopholing (ICLR 2026), using a deterministic latent pathway to solve the issue of token beliefs collapsing into one-hot vectors during discrete diffusion sampling.

Original post →

DiffusionGemma Report: Parallel 256-Token Generation Breaks AR Bottleneck

More from Models