New on-policy distillation method lets student models match or exceed their teachers across four pairs
_akhaliq · x · 2026-10-02
The paper "The Teacher Is a Direction, Not a Destination" introduces a new on-policy distillation method that extrapolates RL-induced representation residuals, letting students continue along the direction their teacher points to rather than stopping at the teacher's ceiling.
Key points:
- Students match or exceed their teachers across four model pairs
- Code and checkpoints are planned for release
This challenges the standard distillation assumption that students can only approach teacher performance.
More from Research
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Open-source TotalSegmentator spots a 42mm dilated aorta on a routine chest CT — MaziyarPanahi · 2026-11-16
- Tivadar Danka Publishes 4,000-Word Visual Guide to Graph Theory for Machine Learning — TivadarDanka · 2026-11-16
- METR time-horizon chart updated: doubling every 1.7 months since Opus 4.5, 'Model 4' ~38h — ben_j_todd · 2026-11-16
- GTAgentClinic: one-line install, plug in Claude Code or Codex for a 3D hospital shift — Michael_D_Moor · 2026-11-16