New Paper: On-Policy Reverse Distillation Lets Stronger Students Surpass Weak Teachers
algo_diver · x · 2026-09-10
An arXiv paper (with Aaron Courville among the authors) tackles weak-to-strong generalization for successive model generations, asking whether frontier-scale post-training can be accelerated by reusing previous, weaker models instead of retraining from scratch.
Key points:
- Conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student.
- The proposed On-Policy Reverse Distillation (OPRD) instead measures the teacher's policy shift relative to its reference policy on student rollouts, and amplifies the verifier-supported component of the student's policy gradient along that direction.
- This preserves the stationary points of policy optimization while accelerating learning beyond the teacher.
- In both successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches; response-style analysis shows students stay closer to verifier-RL-trained models than to their weak teachers.
More from Models
- DeepSeek's V4.1 Flash reworks the Transformer again: asymmetric encoder-decoder and CSA2 — evijit · 2026-09-10
- DeepSeek returns to encoder-decoder with new open architecture changes — evijit · 2026-09-10
- 100M output tokens for $60: DeepSeek off-peak pricing undercuts Opus 5 by 40x — airesearch12 · 2026-09-10
- Bug Hunt Bench ranks frontier coding models on 105 real planted bugs, with 200x cost spread — PawelHuryn · 2026-09-10
- Dev on both Claude and ChatGPT $20 plans hits limits, calls pricing convoluted — MaxLenormand · 2026-09-10
- Rumor: DeepSeek trained a 3T-parameter model but held back over serving economics — zephyr_z9 · 2026-09-10