Diffusion LMs can't fix their own bugs — NeurIPS paper shows why and how to fix it
burny_tech · x · 2026-09-27
A NeurIPS 2026 paper (arXiv:2512.15596, "Corrective Diffusion Language Models") explains why diffusion language models (DLMs) can't reliably revise their own bugs: they're as confident in an erroneous token as in a correct one.
Key finding
- Standard masked-diffusion loss only supervises [MASK] positions; visible tokens never receive gradients, even when wrong, so model confidence can't separate right from wrong.
- Confidence-based remasking then overwrites correct tokens while leaving the actual error in place.
Fix and benchmark
- The authors propose a correction-oriented post-training principle that explicitly supervises visible incorrect tokens, enabling discriminative confidence and targeted refinement — a single change to the training objective.
- They introduce the Code Revision Benchmark (CRB): type-preserving corruptions (operators, identifiers, literals) injected into HumanEval/MBPP solutions with a controllable error count, execution-verified — the first executable benchmark for in-place correction in DLMs.
- Models trained this way substantially outperform standard MDLMs, with the largest gains under heavy parallel-decoding uncertainty and iterative refinement.
More from Research
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- Kind ICLR Reviewer Raises Score Despite Unresolved Concerns, Citing the Paper's Value — MarkSchmidtUBC · 2026-11-16
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16