OPD: Roles, Pathologies, and Regulation
Rui Wang · hf · 2026-07-17
The Roles, Pathologies, and Regulation of On-Policy Distillation
This paper systematically investigates the role and failure modes of On-Policy Distillation (OPD) in LLM post-training. The authors' core conclusion is that OPD acts more like an "exploration catalyst"—it pushes the student model toward correct reasoning paths via dense token-level guidance, but doesn't raise the capability ceiling.
Main Findings
- Prompt diversity is more important than per-question sampling frequency
- OPD's effect relies entirely on the quality of the guidance signal
- The key isn't the teacher's size, but whether the signal is reliable
Two Pathologies
- Student-Teacher Mismatch: When the distribution gap between student and teacher is too large, the guidance signal misleads exploration, pushing it in the wrong direction
- Length Exploitation: Token-level objectives are easily exploited by length cheating (e.g., truncation or redundant padding), leading models to explore "bad length patterns" rather than reasoning strategies
Regulation Methods
The authors tested lightweight regulation methods, including:
- Advantage clipping
- Log-scale compression
These methods mitigate length speculation and make distillation more stable. Experiments covering 7 benchmarks show that regulated OPD consistently outperforms OPD variants and RLVR baselines.
More from Research
- Stanford Team Introduces Gigatoken, the World's Fastest Tokenizer — StanfordAILab · 2026-07-22
- Tabul AI launches Metal TreeSHAP to speed up Shapley values on Apple silicon — Scobleizer · 2026-07-22
- Reddit points to OpenAI’s ChatGPT Ads page — EcstaticAsparagus509 · 2026-07-22
- Open-source runtime lets each repo define its own AI code reviewer — ibabufrik · 2026-07-22
- DeepSWE: A New Benchmark for Evaluating AI Coding Agents on Real GitHub Issues — pmz · 2026-07-22
- A Rust space-economy sim runs hundreds of autonomous ships, built with Claude — kalcode · 2026-07-22