OPD: Roles, Pathologies, and Regulation

Rui Wang · hf · 2026-07-17

The Roles, Pathologies, and Regulation of On-Policy Distillation

This paper systematically investigates the role and failure modes of On-Policy Distillation (OPD) in LLM post-training. The authors' core conclusion is that OPD acts more like an "exploration catalyst"—it pushes the student model toward correct reasoning paths via dense token-level guidance, but doesn't raise the capability ceiling.

Main Findings

Prompt diversity is more important than per-question sampling frequency
OPD's effect relies entirely on the quality of the guidance signal
The key isn't the teacher's size, but whether the signal is reliable

Two Pathologies

Student-Teacher Mismatch: When the distribution gap between student and teacher is too large, the guidance signal misleads exploration, pushing it in the wrong direction
Length Exploitation: Token-level objectives are easily exploited by length cheating (e.g., truncation or redundant padding), leading models to explore "bad length patterns" rather than reasoning strategies

Regulation Methods

The authors tested lightweight regulation methods, including:

Advantage clipping
Log-scale compression

These methods mitigate length speculation and make distillation more stable. Experiments covering 7 benchmarks show that regulated OPD consistently outperforms OPD variants and RLVR baselines.

Original post →

OPD: Roles, Pathologies, and Regulation

More from Research