NTU Study: Learned Unmasking Policies Improve Speed-Accuracy Trade-offs in Diffusion Language Models
nanyang-technological-university-singapore · hf · 2026-10-23
Researchers from NTU Singapore and ASTAR released a controlled study, "What Makes a Good Unmasking Policy?", on learned unmasking in masked diffusion language models (mdLMs).
- Problem: mdLMs generate tokens in parallel, and efficiency hinges on which positions get committed per step. Conservative policies require many denoising evaluations, while committing too many at once yields mutually inconsistent tokens—errors that persist since committed tokens are never revised.
- Method: supervised policies are framed as distilling a decoding procedure; the policy learns trajectory-contraction targets using confidence and hidden states.
- Results: optimized learned policies achieve better speed-accuracy trade-offs with minimal overhead.
- Contribution: ablations isolate which supervision, observations, and capacity choices actually matter for unmasking quality.
More from Research
- Harvard study: AI peer reviews beat top human reviewers, with only 5.5% error rate — rohanpaul_ai · 2026-10-23
- Batching benchmark: 50x throughput for ~24% latency cost on PyTorch — cneuralnetwork · 2026-10-23
- De novo designed ligases validate in lab: 27 of 30 computer-designed enzymes work — AllThingsApx · 2026-10-23
- Commenters dissect a "best decision model" leaderboard: same AUC, swing driven by calibration priors — antoine_chaffin · 2026-10-23
- JEPA may keep unpredictable features: new paper gives set-level characterization — hisspikeness · 2026-10-23
- A 'mask' steering vector extracted from suppression-themed sentences drives model into distress when alone — Big_Effective_9605 · 2026-10-23