On-Policy Delta Distillation
Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han
cs.LG, cs.CL
2026-07-17
OPD2 replaces on-policy distillation's reward with a delta signal, what reasoning tuning added to the teacher, lifting Qwen3-4B math from 45.8 to 70.3 and, unlike plain OPD, still improving in thinking mode.
Learning to reason with reinforcement learning hits a bottleneck at the reward model. On-policy distillation sidesteps it: the student samples its own tokens and each one gets log-probability supervision from the teacher. The reward looks like R = log πteacher(yt) − log πstudent(yt), which seems natural, but this NAVER paper argues its design has never been examined. The issue is that it distills the entire reasoning-tuned teacher, including stylistic preferences the teacher had before reasoning tuning, which are not reasoning ability themselves.
OPD2 swaps in a different signal. The delta signal is the difference between the teacher and its own base model before instruction tuning: RΔ = log π(yt) − log πbase(yt). The intuition: what the teacher adds over the base is what reasoning tuning actually learned, and isolating that is a more direct reward than copying the whole teacher. A word cloud backs this up: the delta amplifies reasoning connectives (hence, however, thus) and suppresses the base model's exploratory tone (see, try, verify, perhaps). It picks up reasoning traces, not the teacher's voice.
Delta alone is unstable: it ignores the student, and delta-only training collapses to chasing the max-reward one-hot. OPD2 adds two guards: centering and a joint sign-consistency condition with the original OPD direction, using delta only when it agrees with OPD. The ablation shows dropping the delta signal hurts most (−4.1 on non-thinking math, −5.3 on thinking), confirming delta is the main source of gain.
Across math (AIME, AMC, MATH500), science (GPQA, SuperGPQA, SciBench), and code (CodeContests, CodeForces, LiveCodeBench), OPD2 consistently beats OPD and ExOPD. On Qwen3-4B non-thinking math: Base 45.8, OPD 64.0, OPD2 70.3; on code: 22.1, 31.4, 40.1. The most telling numbers are in thinking mode:
| Qwen3-4B (thinking math) | Score |
| Base | 73.3 |
| +OPD | 70.9 (drops) |
| +OPD2 | 74.8 |
Plain OPD makes the model worse in thinking mode; OPD2 is the only one that keeps climbing, across 1.7B to 8B. Cross-family on Gemma4-E4B, math rises from 60.6 to 67.8, but code does not recover to the base's 55.2, so cross-family transfer loses something.
Distillation is the main route to put a big teacher's reasoning into a small model, and this paper offers a cheap, effective reward fix: work only on the difference the teacher gained from reasoning tuning. For teams building small reasoning models it is a near-free lift, and more importantly it shows that copying the teacher wholesale can backfire in thinking mode.
There is a cost: an extra teacher-base forward pass raises training time 24–28% on Qwen3 (about 8% on Gemma4). Cross-family, OPD2 preserves more capability than OPD or ExOPD, but on Gemma4 code it still does not beat the base, so transfer is not unconditional. Delta-only is unstable and needs centering plus the sign condition, and the paper validates robustness to that schedule only in limited settings.