Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
Zhaoyi Li, Deyang Kong, Yuan Wei, Evan Yang, Ranran Shen, Mahardika Krisna Ihsani, Ming Yang, Wei Zhang, Chuan Hao, Jian Yang, Ran Tao, Bryan Dai, Shikun Zhang, Wei Ye, Ying Wei, Defu Lian
cs.CL
2026-08-17
OPD barely cares about problem difficulty; same-origin teachers transfer across language, horizon, and domain, while MOPD mix ratios create a capability seesaw.
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision, which cuts exposure bias. Most papers report gains in one domain on benchmarks close to the training set, so local fitting and broader policy transfer stay mixed together. Multi-teacher OPD (MOPD) usually routes each prompt to a domain expert and treats that routing as isolation.
A team from USTC, Peking University, IQuest, and others varies one generalization factor at a time: in-domain difficulty, language, and reasoning horizon, then cross-domain transfer, then teacher mixtures.
The objective is reverse KL between student and teacher, estimated with a sampled-token k1 approximation and written as a policy gradient: a token is reinforced when the teacher assigns it higher probability than the student, and suppressed otherwise. Same-origin means teacher and student share a base (typically an SFT student and an RL teacher from the same checkpoint). Cross-origin means different bases. Math training defaults to BigMath and is scored on AMC2023, MATH-500, AIME2025/2026, BeyondAIME, and OlymMATH-Hard; Chinese olympiad sets probe language shift, and composed Horizon subsets probe long-horizon reasoning. Code uses DeepCoder / LiveCodeBench; science uses TextbookReasoning and SCP-116K / GPQA-Diamond; instruction following uses Nemotron-IF / IF-Eval.
Three 25K-problem slices of BigMath, labeled by teacher pass-rate 1, 0, and random, converge to nearly the same accuracy. Grade-school GSM8K and DeepMath-103K items with difficulty above 8 still recover more than 80% of the BigMath-random OPD gain, and the two finals differ by under two points. Dropping problems the student already solves helps a little:
| Pair | Filter | Six-benchmark average |
| Polaris-7B → DS-distill-1.5B | none / drop pass-rate=1 | 41.4% / 42.0% (+0.6) |
| Light-R1-14B → DS-distill-7B | none / drop pass-rate=1 | 52.4% / 52.8% (+0.4) |
Keeping only fully solved or fully unsolved problems does not help. Training on English short-horizon math still lifts Chinese and long-horizon math. A stronger cross-origin teacher (Light-R1-14B) barely moves a 7B student on long-horizon items; a weaker same-origin teacher (Polaris-7B) pulls the student near the teacher's own level.
Cross-domain transfer is sharper. With a same-origin teacher, math prompts move LiveCodeBench and GPQA toward the teacher; code or science prompts move math, including Chinese and long-horizon, the other way. When the teacher is weaker than the student on science (JustRL-1.5B), either prompt domain drags GPQA down. Cross-origin runs keep a clear in-domain gap: the LiveCodeBench curve from code prompts sits well above the curve from math prompts.
Prompt routing does not contain that spillover in MOPD. JustRL (stronger math) and Nemotron (stronger science/IF) teach DS-distill-1.5B with a fixed total prompt count and a varying mix:
| JustRL/Nemotron | BeyondAIME+OlymMATH average |
| 25/8 | 27.1% |
| 1/1 | 26.3% |
| 2/25 | 25.1% |
Raising JustRL's share pulls GPQA, LiveCodeBench, and IF-Eval toward JustRL's lower scores; GPQA can drop about 5 points early in training. Swap the domain labels and the same three benchmarks rise with Nemotron's share. In a cascade, Nemotron on science prompts lifts GPQA, then JustRL on math prompts pulls it back. Top-16 next-token overlap starts higher and keeps rising for same-origin teachers, and stays flat or falls for cross-origin ones. A same-origin teacher also pulls harder in a 1:1 MOPD mix.
Filtering OPD data by whether the teacher can solve the problem is barely worth the cost. The origin match between teacher and student is the knob that actually moves generalization. For a single same-origin teacher, that reach across language, horizon, and domain is useful. With several teachers it becomes a tug-of-war: a weak score on one domain may come from another teacher's cross-domain pull. Routing is not a capability fence, and it is not a safety fence either.
The study stays on reasoning models in math, code, science, and instruction following. Multimodal, tool-using, and interactive agent settings are untested. MOPD uses two complementary teachers and fixed domain routing; larger expert pools and adaptive routers are open. The same-origin story stops at top-k overlap, with no finer representation analysis. "Unsolved problems still help" is a token-level KL result; offline answer distillation may not behave the same way.