Calibrating Teacher--Student Discrepancy for On-Policy Distillation
Qiangqiang He, Jin Li, MingCai Chen
cs.AI
2026-09-18
Cal-OPD probes teacher self-jitter, keeps 52–65% of the OPD signal, and beats standard OPD by 2.3 and 3.1 points on two Qwen3 pairs.
On-policy distillation (OPD) trains a student on its own sampled traces, using the token-level log-likelihood gap between a frozen teacher and the student as dense supervision. That is denser than outcome-only RLVR and avoids the off-policy mismatch of a fixed distillation corpus. The usual assumption is that the teacher’s score on each student token is a trustworthy target.
It is not equally trustworthy. Hold the problem and the student trace fixed, add a bit of extra context to the teacher, and the teacher log-likelihood on the same token moves. The paper calls this Teacher Self-Deviation (TSD). On Qwen3-8B teaching Qwen3-1.7B over about 6,528 DAPO questions and 60 million tokens, even task-agnostic instructions such as “reason carefully” versus “answer quickly” produce significant TSD on about 29.8% of tokens. Feeding a gold answer or a reference solution keeps almost all of those positions (retention 98.2%–98.6%) and expands the affected set to 39.6%.
TSD also barely cares about polarity. Flipping answer correctness still yields 88.7% directional agreement and a 74.7% shared-deviation ratio. High-TSD surface forms such as maybe, however, therefore, consider fire on more than 89% of occurrences; digits, √, θ, and frac stay below 9.3%. If the student swallows the raw teacher–student gap, a large slice of what it copies is the teacher’s stylistic jitter.
Privileged-OPD, which conditions the teacher on a reference solution and distills that distribution, amplifies the same jitter.
Cal-OPD never distills the privileged context into the student. It uses that context only to measure how far the teacher moves.
For each token on the student trace, the teacher is probed with a contrasting pair. The default pair is evaluative feedback: a gold verifier calling the trace rigorous versus calling it unreliable, with no leaked solution. The max upward and downward log-likelihood shifts define an empirical TSD band, stretched by a relaxation factor λ=5 into an interval [L, U]. If the student log-likelihood sits inside the interval, the advantage is zero. Below L it is pushed only to the lower edge; above U only to the upper edge. The original OPD gap therefore splits into a TSD-explained piece and a calibrated residual. Only the residual is optimized.
Training runs in verl on 8 H20 GPUs (4 student, 4 teacher), 100 steps, 256 traces per step, learning rate 1e-6. Evaluation is Avg@16 at temperature 0.6.
Average of six math benchmarks, two Qwen3 thinking pairs:
| Setup | Student | Standard OPD | Privileged-OPD | Cal-OPD |
| 4B-Thinking → 1.7B | 49.2 | 50.8 | 49.3 | 53.1 |
| 30B-A3B-Thinking → 4B | 66.6 | 65.9 | 64.0 | 69.0 |
Gains versus the student are +3.9 and +2.4; versus standard OPD, +2.3 and +3.1. Cal-OPD wins 7 of 12 benchmark–setup cells. ExOPD, EOPD, and Uni-OPD stall at 50.5–51.9 and 67.1–67.6. Standard OPD actually drops the larger student from 66.6 to 65.9. Privileged-OPD is the worst distiller in both regimes.
During training the retained OPD gap falls from about 65% to 52%, and the zero-advantage token share rises from 27% to 34%. Standard OPD stretches mean response length from about 9.8K to 11.8K tokens; Cal-OPD finishes at 9.3K. On the small pair that makes Cal-OPD faster per step (14.90 vs 18.80 minutes), because shorter traces outweigh two extra teacher forwards. On 30B→4B it is slower: 48.76 vs 31.26 minutes per step.
The probe set matters. Evaluative feedback is best (average 53.1). Solution-level privilege is worst (49.0), retaining only about 20% of the signal and pushing the student into a high-entropy, short-response regime. λ=5 is the sweet spot; a wider band cuts useful supervision. Retention-matched attenuation baselines do not catch up, so the gain is about which gap is removed, not about making the gradient smaller.
OPD is a common shortcut beside RLVR for reasoning post-training. The practical warning is simple: a teacher–student gap is not a pure capability gap. Surface-token jitter gets copied as knowledge, and a privileged teacher makes that worse. Privileged information is useful as a measuring stick for the teacher, not as a second teacher to imitate.
This is a cleaning step on the OPD signal, not a new distillation objective. Entropy-aware, exploration, and reward-extrapolation variants all leave the “should this gap be learned” question untouched. On the small pair, shorter traces even pay for the extra forwards.
All numbers are math reasoning on two Qwen3 thinking pairs. No code, agents, or non-thinking models. The TSD band is a finite-probe approximation; λ and the prompt pair are hyperparameters. Solution-level privilege over-filters, which means “jitter” and “real knowledge” are entangled once the context is rich. Extra teacher forwards dominate when the teacher is a 30B MoE. The paper has no dedicated limitations section, so transfer across domains and model families is unverified.