On-policy distillation explained: teachers grade every token of student trajectories
helloiamleonie · x · 2026-09-25
Leonie walks through on-policy distillation (OPD): the student's own trajectory is passed to a teacher model, which grades each token by its own probabilities — measuring how 'surprised' it is at each step. Compared to sparse verifiable rewards, this gives the student a much denser, richer learning signal.
More from Research
- Full paper: active learning cuts surgical video annotation effort by ~50% — santiviquez · 2026-11-11
- Active learning with weak supervision cuts surgical video annotation effort by ~50% — Laparoscopes · 2026-11-11
- Active Learning with Weak Supervision Cuts Surgical Video Annotation Effort by 50%, Penn Study Finds — Laparoscopes · 2026-11-11
- Tuva Labs takes 1st on ARC-AGI-2, 2nd on ARC-AGI-3 in ARC Prize 2026 — GregKamradt · 2026-11-11
- Sakana AI × Pathway Post-Transformers lecture on AI and the brain now online — kaixhin · 2026-11-11
- Defending METR: public data can't second-guess its AI speedup findings — peterwildeford · 2026-11-11