Direct On-Policy Distillation: Weak-to-Strong Transfer
BytedTsinghua-SIA · hf · 2026-07-14
The authors propose Direct On-Policy Distillation, using policy shifts during the reinforcement learning phase as an implicit reward signal to directly transfer improvements learned by a small model during RL to a larger model.
Core Idea
Traditional approaches often require running expensive RL from scratch on the target large model. This work leverages policy shifts during the "on-policy" phase to approximate which behaviors are better, thereby transferring the small model's gains to the large model without repeating the full RL process.
Goals
The method focuses on:
- Scaling training more efficiently
- Reducing the cost of re-running RL on the target model
- Supporting weak-to-strong generalization
Related event: New On-Policy Distillation Enables Weak-to-Strong Generalization(3 posts)→
More from Research
- Open-source runtime lets each repo define its own AI code reviewer — ibabufrik · 2026-07-22
- OpenAI and Apollo Research introduce Contrastive SDF to measure reward-seeking — OpenAI · 2026-07-22
- NVIDIA says to tune the harness before tuning the model with LangChain — NVIDIAAI · 2026-07-22
- The Thimble and the Waterfall: AI's Data Bottleneck and Feedback Loops — dyamins · 2026-07-22
- NVIDIA shows 22 SIGGRAPH papers and Omniverse tools for robot simulation — facontidavide · 2026-07-22
- Building a Knowledge Graph Without a Graph DB: 1000x Cheaper Than GraphRAG — TheRedfather · 2026-07-22