MultiMDM: multi-mask diffusion LMs draft before writing for few-step generation
QuanquanGu · x · 2026-09-05
A UCLA team (Quanquan Gu, Lexing Ying et al.) presents MultiMDM at COLM 2026, addressing why masked diffusion models (MDMs) struggle at few-step generation: all forward trajectories collapse to one fully masked state, leaving no terminal entropy for consistency-style stepping.
Key ideas:
- Preserve masking structure: each clean token is first pushed toward a designated mask, then gradually mixes over the mask set
- This gives the backward process a drafting capability — typed masks sketch the sentence before resolving into tokens, so [MASK] carries a rough plan for its word
- A closed-form ELBO objective supports continual training from pretrained MDMs
- A purely discrete-state consistency distillation scheme with shared-Gumbel coupling reduces pathwise entropy
Experiments on pretraining and distillation show MultiMDM provides an effective foundation for principled few-step generation.
More from Research
- VeriPhy: agentic physical reasoning framework for world model evaluation — Wenzhuo Xu · 2026-09-05
- TRACES agent benchmark grades live execution loops, not answers — SucceededMind · 2026-09-05
- Pedro Domingos quips: 'new idea' called RNNs will power next-gen LLMs — pmddomingos · 2026-09-05
- LoRA Creator Edward Hu Publishes Guide on Post-Training Open-Source Models with RL — iamrobotbear · 2026-09-05
- Amid CoT monitoring buzz, one video offers a glimpse into how LLMs actually think — kastnerkyle · 2026-09-05
- Multi-agent debate can make models dumber: ICML paper identifies sycophancy failure modes — ghadfield · 2026-09-05