Discrete Diffusion vs BERT Masking Objectives
nandofioretto · x · 2026-07-13
The masked / absorbing-state setting in discrete diffusion is essentially very much like putting BERT's masked LM objective into a loop: first train the model to "fill in the blanks", and then during generation, gradually sample by repeatedly unmasking.
The key benefits of this approach are:
- Tokens can be decoded in any order and in parallel
- No need to generate strictly from left to right like autoregressive models
- The generation process is completed through iterative unmasking, making it more suitable for parallel inference
The author summarizes it in one sentence: there is actually only a layer of "loop iteration" between discrete diffusion and BERT-style masked modeling.
More from Research
- Linear Digressions returns with a new season of audio essays on AI agents — ChrisGPotts · 2026-07-21
- Async OPD distillation doubles throughput while matching synchronous math accuracy — _lewtun · 2026-07-21
- A forecasting lesson on why R-squared alone led to overfitting and worse predictions — mdancho84 · 2026-07-21
- Google DeepMind’s Project Genie talk shows how creatives feed into model research — alexanderchen · 2026-07-21
- Nat Lambert says RL distillation does not use the strongest models as teachers — natolambert · 2026-07-21
- Thread claims GPT-5.6 Sol helped build a new counterexample factory for the Jacobian conjecture — LucaAmb · 2026-07-21