Direct-OPD: Transferring RL Policy Shifts from Small to Large Models
_lewtun · x · 2026-08-12
The HF journal club discussed Direct On-Policy Distillation (Direct-OPD), a paper from a joint Tsinghua and ByteDance lab.
The method tackles the high cost of running Reinforcement Learning (RL) on large models. The core idea is to run expensive RL on a smaller model, extract the policy shift (the difference between pre-RL and post-RL checkpoints), and distill this shift into a larger, already stronger model.
Unlike naive distillation (which inherits the small model's limitations), transferring this RL-induced policy shift yields strong results. On AIME24, a 7B student model improved its score from 56.7 to 63.1 by absorbing the policy shift from a 1.5B teacher pair.
Related event: Tsinghua and ByteDance Propose Direct-OPD for Model Distillation(3 posts)→
More from Research
- Gemma 4 QAT Shows Significant Improvement in KV Cache Quantization, KLD Benchmarks Reveal — Anbeeld · 2026-08-12
- Recommended: 303-Page Comprehensive Survey on Code Models and Agents — mdancho84 · 2026-08-12
- Multi-Agent Debate Can Cut Hallucinations if Set Up Right — mdancho84 · 2026-08-12
- Code Models Are Insecure by Default; MoE is More Fragile Than Dense Models During SFT — mdancho84 · 2026-08-12
- Coding Scaling Laws Vary by Language; Python's Dynamic Typing Causes Negative Transfer — mdancho84 · 2026-08-12
- 303-Page Survey: RLVR Helps Small Models Punch Above Their Weight in Coding — mdancho84 · 2026-08-12