On the Off-Policy Teacher in On-Policy Distillation
Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Blöbaum, Purak Jain
cs.LG, cs.AI, cs.CL
2026-09-30
OPD leaves the teacher supervising off-policy student prefixes. SCOUT retrains it with RL on those prefixes, adding 1.2-2.6 math and 3.1 code points.
On-policy distillation (OPD) trains a student on trajectories sampled from its own policy, with a stronger teacher providing next-token distributions at every position. Dense supervision, on states the student actually visits. But the arrangement is on-policy for only one side. The trajectories are off-policy for the teacher: it was optimized to continue from prefixes it would write itself, yet in OPD it must score prefixes written by the student. Differences in reasoning style and intermediate errors accumulate token by token, and the longer the student prefix, the further it drifts from anything the teacher would have produced.
The paper measures this directly. With Qwen3-1.7B (non-thinking) as student and Qwen3-4B-Instruct-2507 as teacher on AIME 2025, student responses are truncated at 10% to 90%, and the teacher continues each prefix four times. Continuation accuracy falls as the prefix grows. Teacher entropy on its own text declines over a response, while entropy on student prefixes stays high, leaving a persistent gap. The continuations sit well inside a 16,384-token budget, so running out of length is not the cause.
Existing fixes all keep the teacher frozen and adjust how its supervision is consumed: ESR distills only on an early prefix, Prune-OPD drops supervision where teacher-student compatibility is low, Relay-OPD lets the teacher take over mid-rollout, TrOPD restricts updates to reliable regions. SCOUT asks the opposite question: why not train the teacher to be good at the states the student actually visits?
SCOUT (Student-COnditioned Updates of the Teacher) leaves the student-side OPD objective untouched and adds periodic teacher adaptation.
Two design choices are deliberate. The student policy drifts during training, so its context distribution drifts too and the teacher must keep catching up, though not every step: ablations show f=1, 5, and 10 land within noise of each other (Avg@6 of 51.3, 52.5, 51.5), and only f=20 falls back toward frozen-teacher level. Also, the student-prefix ratio grows linearly over training. Figure 1 showed long prefixes are the hard case, so training starts on short ones and stretches them as the teacher adapts.
Four settings, and SCOUT beats OPD with a frozen teacher on mean accuracy in all of them. Each configuration is trained with three random seeds and evaluated 4 to 32 times per benchmark.
| Setting | Frozen-teacher OPD | SCOUT | Gain |
| Qwen3-4B→1.7B, math mean (6 benchmarks) | 49.2 | 51.4 | +2.2 |
| Qwen3-4B→1.7B, code mean (3 benchmarks) | 56.6 | 59.7 | +3.1 |
| Qwen3-8B-DAPO→1.7B, math mean | 49.0 | 51.6 | +2.6 |
| Skywork-OR1-7B→R1-Distill-1.5B, math mean | 52.4 | 53.6 | +1.2 |
No competing OPD variant beats SCOUT on mean accuracy in any setting, and each has a failure mode: Relay-OPD collapses to 12.1 on code (frequent syntax errors), Prune-OPD drops to 36.8 in the cross-family setting, and GRPO training there collapses outright to 22.6. Methods that modify the student's rollouts turn out sensitive to domain and model family; adapting the teacher held up in all four.
The decisive control is "OPD + Teacher GRPO": identical RL on the teacher, but rollouts start from the original problem instead of a student prefix. It gets 48.8 (4B math, below OPD's 49.2), 57.0 (4B code), and 50.8 (8B math), versus SCOUT's 51.4 / 59.7 / 51.6. Extra teacher training alone does not buy the gains; conditioning on student prefixes does.
Two more results. Under matched compute (SCOUT for one epoch versus OPD for three on the same data), OPD plateaus quickly while SCOUT keeps improving. And SCOUT stacks with OPTR, the trust-region component of TrOPD: math goes from 49.8 to 52.3 and code from 59.3 to 60.5, so teacher adaptation and loss-level fixes are complementary.
Most OPD research tunes how a fixed teacher's signal is consumed. SCOUT establishes teacher adaptation as a separate, stackable optimization axis, with a control experiment that isolates the mechanism. For teams distilling small models on verifiable tasks, it is a directly applicable add-on worth 1.2 to 2.6 points in every setting tested. The price is real: the teacher must be trainable and the task must support verifiable rewards.
The authors concede that in the compute-matched comparison the extra OPD epochs reused the same data, so fresh data might close part of the gap.
Unverified or undiscussed: teacher-side RL needs outcome rewards, so the evidence covers only math and code, with nothing for open-ended tasks; the cost of co-training the teacher is never quantified; scale stops at an 8B teacher and a 1.7B student, and the motivating off-policy-teacher measurement was run on a single benchmark (AIME 2025).
One observation the paper never discusses: in the 4B→1.7B code setting, plain GRPO averages 62.2 against SCOUT's 59.7 (on math, GRPO's 50.3 trails SCOUT's 51.4). The net case for distillation is concentrated in math; on code, plain RL holds its own.