On-policy distillation gains come from suppressing low-prob tokens, not the teacher
burny_tech · x · 2026-09-03
The paper "Does On-Policy Distillation Really Distill?" (Yi Ding, Ruqi Zhang) quantitatively dissects where on-policy distillation (OPD) gains actually come from, with surprising results:
- Teacher signal is noisy: teachers score student-generated trajectories that are inherently off-policy, and supervision noise actually increases with teacher scale.
- Students are insensitive to the noise: removing noisy supervision yields comparable performance — the teacher barely distills anything.
- Real source of gains: learning concentrates on low log-probability tokens; a single fixed negative advantage matches teacher-provided scores. OPD largely works by suppressing tail tokens.
The authors propose On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages: stronger signals at high-entropy positions, tail-token suppression, and even redistribution of probability mass among head tokens.
Results: versus base Qwen3-1.7B, OPSA improves Avg@32 on AIME24 by 35.41 points (263% relative gain), more than doubles Pass@32 across three benchmarks, and beats OPD by 16.77 points on AIME24 Avg@32, generalizing across model families and tasks.
More from Research
- Terence Tao on 'War Footing' as OpenAI Claims Navier-Stokes Solve Shakes Mathematics — AlexTensor · 2026-10-18
- Univ. of Tokyo paper: why your recorded voice feels eerie — a vocal uncanny valley — ssh4net · 2026-10-18
- MIT's SANDO Drone Planner Avoids Every Moving Obstacle Across 12 Real Flights — Olivier__OG · 2026-10-18
- No-backprop pretraining speedup of 2.3x claimed, inspired by mech interp essay — CatAstro_Piyush · 2026-10-18
- Models Can Be Trained to Sense Activation Probes and Generalize Across GPUs Zero-Shot — Sauers_ · 2026-10-18
- Simulating a Turing machine with addition and multiplication by encoding the tape as a number — jessi_cata · 2026-10-18