On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation
Yichen Wang, Yufa Zhou, Siqi Zhu, Haozheng Luo, Jingzhi Bao, Zhichao Liu, Zijun Wei
cs.LG
2026-11-04
Qwen3 experiments show on-policy distillation matches teachers in-distribution, yet OOD scores fall up to 16.25% below the base and composed tasks trail best teachers by 18-24%.
Frontier post-training pipelines often split the work: train separate specialist teachers for math, code, agent use, and long context, then consolidate their gains into a single student. Multi-teacher on-policy distillation (MOPD) is an increasingly common consolidation step, and prior reports describe near-perfect distillation on the training distribution.
The catch is that those sign-offs are almost all in-distribution (IND). Consolidation is supposed to deliver the teacher's gains, keep the base model's existing capabilities, and stay usable beyond the supervised tasks. Near-perfect IND scores guarantee neither of the last two. A team spanning ByteDance and several US universities shows all three can coexist: near-perfect IND distillation alongside OOD forgetting and missing compositional ability.
OPD trains a student on trajectories it samples itself, supervised by the teacher's next-token distributions, minimizing reverse KL at every position. It borrows RL's on-policy state distribution and distillation's dense token-level signal, which is why it converges fast. MOPD sums OPD objectives across domains with fixed mixture weights, one teacher per domain. The evaluation has three layers:
Two fixes follow from the analysis: geometry-based patching, which aligns the OPD parameter update to the RL teacher's update via orthogonal Procrustes and subtracts the unmatched residual after distillation; and base-preserving replay, which mixes in prompts supervised by the base model itself.
The synthetic study sets the signal: distillation ratios of 96-104% on the pretraining range and 93-96% on the post-training range fall to 76-86% on held-out 15-17 operations.
| Setting | Metric | Result |
| Single-teacher OPD (3 math datasets) | IND distillation rate | 97.10% of teacher |
| Same | OOD vs. base | -8.56 points (-10.78%) |
| MOPD (pairwise + 3-teacher mixes) | OOD vs. base | -11.18 points (-14.07%) avg |
| Qwen3-32B → 8B external teacher | IND / OOD | 99.34% / still -2.96 points |
| Three compositional settings | vs. best teacher | -18.19% to -24.34% |
Two qualifiers. Swapping in a stronger external teacher does not close the gap, even though the 32B teacher beats the base on OOD evaluations. And (M)OPD still forgets less than its RL counterparts: 25.09% less than the self-RL teachers in the single-teacher setting, 28.92% less for MOPD than mixed RL. Distillation is not the source of forgetting; it just fails to repair it.
The compositional numbers are the sharpest. In math+long context, MOPD lifts LongBench-v2 from 40.50 to 56.20 and GSM8K from 83.40 to 95.53, yet composed tasks sit 24.3% below the best teacher. Code+long context is more extreme: all three component benchmarks improve while LongBench-v2 Code stays flat at the base's 44.00, against teachers at 50.00 and 58.00.
The mechanism work pins down two bottlenecks, teacher capability and data coverage. With the 32B teacher plus 10% OpenR1 mixed into GSM8K, the student beats the base on all five benchmarks, matches the teacher's AIME24/25 pass@8 (86.67/83.33), and hits 100.00 on AMC23 above the teacher's 95.00. Update geometry offers a lens: OPD's update norm is 55.4% of RL's (0.154 vs. 0.278), the top-64 singular directions carry 69.0% of update energy versus 39.9% for RL, and MOPD updates project far better onto single-teacher OPD spans (R² of 11.0-18.6%) than mixed RL onto single-RL spans (under 0.24%).
For the fixes: Procrustes patching at α=0.40 raises AIME25 avg@8 from 60.83 to 65.00, within 0.83 points of the base, holding GSM8K at 93.48. Base-preserving replay improves five of eight OOD benchmarks, adds 6.67 points on AIME25 avg@8 to 67.50 (above both base and RL teacher), and costs 0.18 points on GSM8K.
This lands directly on how consolidation gets accepted. If the report card only scores the training distribution, a model that failed to consolidate can graduate with near-perfect marks. Teams running MOPD should put OOD retention and composed tasks in the gate; those numbers are decoupled from IND distillation rate.
The fixes are cheap: mixing in 10% broader data, letting the base model supervise part of the batch, or running one post-hoc alignment patch are drop-in changes. The distillation objective lacks an explicit base-preserving term, and adding one pays.
Author-stated: students fixed at Qwen3-8B and the largest external teacher at Qwen3-32B, so capacity and model-family dependence are unexplored (a small Qwen3-235B-A22B run suggests the limitation is not minor at scale); OOD labels are relative to post-training data, not pretraining exposure; MOPD and mixed RL were compared out-of-the-box without tuning mixture weights or curricula; no repeated training runs, only four evaluation sampling seeds on a 10% subset; and when a task is directly learnable from verifier feedback, mixed RL with SFT warmup overtakes MOPD, so that advantage is conditional. The link between update geometry and generalization is explicitly a hypothesis without a causal test.
Two more caveats from a close read. The "broader data eliminates forgetting" run adds OpenR1, which itself covers harder problems, so what disappeared may be the distribution shift rather than forgetting; AMC23 above the teacher is the only hard evidence of generalizing beyond the teacher. And patching needs a separately trained RL teacher's weights as its alignment target, an artifact a distillation-only pipeline may not have.