DeepMind paper proves sequence-level supervised losses for autoregressive models can be fundamentally flawed
On November 17, Google DeepMind researcher Mathieu Blondel and colleagues published a paper on arXiv, "On the Difficulty of Sequence-Level Supervised Learning for Autoregressive Models," systematically demonstrating that sequence-level supervised training of autoregressive models faces fundamental difficulties.
Confirmed
- Motivation: teacher forcing training introduces exposure bias, and sequence-level supervision is a natural alternative — but the authors prove it has fundamental flaws
- Two natural constructions of sequence-level losses — based on greedy decoding and on autoregressive sampling — are both shown to have unbounded lower bounds, so directly optimizing them makes the model diverge
- The two losses can be related to each other via the autoregressive Gumbel trick
- Since the pathology stems from the sampling process's dependence on model parameters, the authors analyze the effect of applying a stop gradient to the sampling, arriving at a theoretically sound "moment matching" update direction, though it has high variance in practice
Why it matters
- The result provides a theoretical explanation for "why you can't simply apply sequence-level losses directly to AR models for gradient training," with direct relevance for training paradigm choices in autoregressive generative models for speech, language, and more. The moment matching direction the authors identify, despite its high variance, also offers a lead for future algorithmic improvements.
All of the above comes from Blondel's own paper-explainer thread on X (m1–m3 are parts of the same thread; m4 and m5 are paraphrases).
2026-11-17 ~ 2026-11-17 · 5 related posts
Primary sources
- [source] Sequence-level supervised learning for AR LMs is fundamentally flawed, Google team shows — mblondel_ml · 2026-11-17
- [source] Both greedy and AR-sampling sequence losses are unbounded below, thread explains — mblondel_ml · 2026-11-17
- [source] Stop-gradient on sampling yields a principled but high-variance update direction — mblondel_ml · 2026-11-17
- DeepMind Paper Shows Sequence-Level Supervised Losses for Autoregressive LLMs Are Fundamentally Flawed — mblondel_ml · 2026-11-17
1 near-duplicate retellings: mblondel_ml