New Theory Paper Shows Sequence-Level On-Policy Distillation Has a Future Agreement Bias
kalyan_kpl · x · 2026-11-20
An arXiv paper theoretically analyzes on-policy distillation (OPD) objectives, answering which tokens get prioritized, in what order, and how efficiently.
Key findings:
- Sequence-level OPD exhibits a future agreement bias: it matches the teacher at later token positions first, while at earlier positions it can favor tokens improbable under the teacher (often undesirable). Myopic token-level objectives avoid this, inducing a reverse learning order
- Divergence choice strongly affects optimizability: forward KL has a well-behaved landscape, while reverse KL and other f-divergences can suffer vanishing gradients even when student and teacher are far apart
- Findings are validated across language model post-training settings
For teams using OPD for distillation or post-training, the paper provides principled guidance on choosing between sequence- vs token-level objectives and which divergence to minimize.
More from Research
- Horvitz and Mitchell Publish 'Scientific Progress in AI: History, Status, and Futures' — ArtificialOther · 2026-11-22
- Interview: Medical AI is only as good as human consensus labels — CurieuxExplorer · 2026-11-22
- Science cover paper runs contextuality tasks on up to 105 qubits of Google's Willow chip — dhruv2038 · 2026-11-22
- Why a P=NP proof would collapse most of modern cryptography — doctorBobG · 2026-11-22
- 10 months after a Reddit question on AI solving Millennium Prize Problems, claims abound — haider1 · 2026-11-22
- Fable 5.2 rewrites model weights so an E2E driving bot learns left-side driving — 4310sy · 2026-11-22