Diffusion LMs can't fix their own bugs — NeurIPS paper shows why and how to fix it
burny_tech · x · 2026-09-27
A NeurIPS 2026 paper (arXiv:2512.15596, "Corrective Diffusion Language Models") explains why diffusion language models (DLMs) can't reliably revise their own bugs: they're as confident in an erroneous token as in a correct one.
Key finding
- Standard masked-diffusion loss only supervises [MASK] positions; visible tokens never receive gradients, even when wrong, so model confidence can't separate right from wrong.
- Confidence-based remasking then overwrites correct tokens while leaving the actual error in place.
Fix and benchmark
- The authors propose a correction-oriented post-training principle that explicitly supervises visible incorrect tokens, enabling discriminative confidence and targeted refinement — a single change to the training objective.
- They introduce the Code Revision Benchmark (CRB): type-preserving corruptions (operators, identifiers, literals) injected into HumanEval/MBPP solutions with a controllable error count, execution-verified — the first executable benchmark for in-place correction in DLMs.
- Models trained this way substantially outperform standard MDLMs, with the largest gains under heavy parallel-decoding uncertainty and iterative refinement.
More from Research
- BrushArena streams its RL training runs live, letting anyone watch paintings evolve — xeophon · 2026-09-27
- Digital Consciousness Model Paper: Evidence Against 2024 LLM Consciousness Is Not Decisive — burny_tech · 2026-09-27
- Xiaomi open-sources RL environments on Hugging Face, potentially worth millions — burny_tech · 2026-09-27
- DYSCO recovers governing equations from noisy high-dim data, accepted at NeurIPS — burny_tech · 2026-09-27
- Solomonoff induction mirrors how intelligence works — but is physically impossible — burny_tech · 2026-09-27
- Xiaomi open-sources 7,000+ RL task environments used to train MiMo — burny_tech · 2026-09-27