Sequence-level supervised learning for AR LMs is fundamentally flawed, Google team shows
mblondel_ml · x · 2026-11-17
Mathieu Blondel and colleagues (Google DeepMind) released an arXiv paper studying whether sequence-level losses can replace teacher forcing to fix exposure bias in autoregressive LMs.
Key findings:
- The two natural constructions—losses based on greedy decoding and on AR sampling—are provably unbounded below; optimizing them makes the model diverge. They are linked via an autoregressive Gumbel trick: the sampling-based loss is the expected greedy loss under Gumbel perturbations.
- The pathology stems from the decoded sequence depending on model parameters. Applying a stop gradient to the sampling yields a "moment matching" direction: not the gradient of any loss, but a provably valid descent direction for the AR negative log-likelihood in function space—though high-variance in practice.
Conclusions: teacher forcing remains the de facto strategy; on-policy distillation better addresses exposure bias but needs a teacher; sequence-level training is still best done with RL plus a reward function or model.
More from Models
- Gemini 4 Carbon cache hit rate averages 85.2%, raising cost-efficiency doubts — mariofilhoml · 2026-11-17
- Claude Code Power User Burns 53% of Weekly Limit in Two Days, Downgrades to Sonnet — Inevitable_Truth_85 · 2026-11-17
- SenseNova Loop 9B Claimed to Match Qwen3 27B at 9B Params, GGUF Coming — MeinDruckerSpinnt · 2026-11-17
- SenseNova-Loop-9B Lands on Hugging Face, Claiming 27B-Class Performance — Marcuss2 · 2026-11-17
- AA-PersonalityBench ranks Gemini 4 Argon most metaphorical model, GPT-3.5 Turbo least — ArtificialAnlys · 2026-11-17
- Counterfactual benchmark shows 11 frontier models fail to grasp programming semantics — jessyjli · 2026-11-17