Study: On-Policy Distillation Works by Suppressing Low-Probability Tokens

A paper examining on-policy distillation finds its gains come mainly from suppressing low-probability tokens in the student's sampling rather than meaningful teacher guidance, suggesting models can self-improve without a teacher.

2026-09-01 ~ 2026-09-02 · 2 related posts

• Study: On-policy distillation works by suppressing low-probability tokens — Purdue · 2026-09-01
• Paper Shows On-Policy Distillation Gains Come from Self-Improvement, Not Teacher Guidance — _akhaliq · 2026-09-02