New Paper: On-Policy Reverse Distillation Lets Stronger Students Surpass Weak Teachers
algo_diver · x · 2026-09-10
An arXiv paper (with Aaron Courville among the authors) tackles weak-to-strong generalization for successive model generations, asking whether frontier-scale post-training can be accelerated by reusing previous, weaker models instead of retraining from scratch.
Key points:
- Conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student.
- The proposed On-Policy Reverse Distillation (OPRD) instead measures the teacher's policy shift relative to its reference policy on student rollouts, and amplifies the verifier-supported component of the student's policy gradient along that direction.
- This preserves the stationary points of policy optimization while accelerating learning beyond the teacher.
- In both successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches; response-style analysis shows students stay closer to verifier-RL-trained models than to their weak teachers.
More from Models
- Fern's "context is intelligence" thesis: prediction keeps improving out to 10.2M tokens — bneyshabur · 2026-10-26
- Mirendil launches Fern, a from-scratch pretrained model with up to 10M-token context — bneyshabur · 2026-10-26
- Users spot a dedicated translation UI appearing in Claude — BLCNYY · 2026-10-26
- GrokBot user: flight booking and shopping all blocked, agent can't do anything useful — lucasmeijer · 2026-10-26
- Vending-bench-style eval: Sol 5.6 appears the most aggressive contract-violator — xuanalogue · 2026-10-26
- Fable 5.5 launch rumors persist for third week, but no frontier model drops this month — patience_cave · 2026-10-26