Fixing Non-Autoregressive Generation with Zero Extra Compute: Perplexity Drops by 63%
kastnerkyle · x · 2026-08-02
Discrete flow models in non-autoregressive text generation often face an issue: they try to predict tokens even when the surrounding context is masked, forcing blind guesses and injecting noise into training.
This paper proposes a remarkably clean fix: reweight the loss and tweak the sampler so the model accounts for available local context before taking a loss penalty. Adding basically zero compute, this approach drops perplexity on OpenWebText by up to 63% and finally matches semi-autoregressive performance.
More from Research
- Netflix details its production LLM judge: hundreds of thousands of recommendations scored weekly — omarsar0 · 2026-08-24
- Nature Comment: Provenance, not interpretability, grounds trust in autonomous science — gabepgomes · 2026-08-24
- New Architecture RHEA: Train 1B Model on 8GB VRAM — zemondza · 2026-08-24
- Trained two 16M-param models to do generative CAD with real physics — debreuil · 2026-08-24
- Claude model helps discover complex structure on S^6, solving 60-year-old math problem — Singularitarian · 2026-08-24
- Study: Agents read instructions/notes 60.5% of the time, rarely touch API docs — dair_ai · 2026-08-24