Study: Stronger Teachers Aren't Always Better Sources for On-Policy Distillation
KnowledgeXLab · hf · 2026-10-12
Zhejiang University, Shanghai AI Lab and collaborators systematically study when relative policy differences transfer in on-policy distillation (OPD), unifying the setup as an implicit reward from the log-probability ratio between a proxy expert and reference, enabling any-to-any transfer across scales, stages, and generations.
Three findings:
- Stronger experts aren't necessarily better supervision sources: differences from task-specific training (RL, SFT) transfer consistently, while scale-induced differences transfer weakly or not at all
- Expert and reference play complementary roles: the expert supplies the capability, the reference filters shared behavior and isolates the relevant residual
- A useful difference doesn't transfer to every target: success requires the update to move the target toward the expert and generalize; raw policy distance doesn't predict transferability, but an appropriate reference can rescue failed transfer
More from Research
- RL scaling papers suggest experimental throughput is the first real bottleneck to RSI — eigenron · 2026-10-12
- SoftwareGym synthesizes 182 app environments, lifting CUA OSWorld Pass@3 to 56.97% — Tian Jin · 2026-10-12
- Qualcomm AI uses entropy and CKA to pick layers, beating EAGLE3 defaults in speculative decoding — Qualcomm-AI-Research · 2026-10-12
- Evolved e-puck robots show communication emerges with information value — uv-mex · 2026-10-12
- PreSTO Speeds Up LLM Power Sampling Up to 2.33x via Predictive Subtree Prefetching — du_yilun · 2026-10-12
- AWS+HSBC paper co-evolves agent graders from just 10 labeled examples, matching ground truth — rohanpaul_ai · 2026-10-12