New Paper: On-Policy Reverse Distillation Lets Stronger Students Surpass Weak Teachers

algo_diver · x · 2026-09-10

An arXiv paper (with Aaron Courville among the authors) tackles weak-to-strong generalization for successive model generations, asking whether frontier-scale post-training can be accelerated by reusing previous, weaker models instead of retraining from scratch.

Key points:

Conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student.
The proposed On-Policy Reverse Distillation (OPRD) instead measures the teacher's policy shift relative to its reference policy on student rollouts, and amplifies the verifier-supported component of the student's policy gradient along that direction.
This preserves the stationary points of policy optimization while accelerating learning beyond the teacher.
In both successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches; response-style analysis shows students stay closer to verifier-RL-trained models than to their weak teachers.

Original post →

New Paper: On-Policy Reverse Distillation Lets Stronger Students Surpass Weak Teachers

More from Models