Weak-to-Strong On-Policy Distillation
Fangxu Yu, Zinan Lin, Xiaodong Liu, Weijia Xu, Michael Xu, Tianyi Zhou, Jianfeng Gao
cs.LG
2026-07-29
Microsoft's W2S-OPD turns the logit gap of two weak models into a proxy teacher; an 8B student beats its 4B expert on math and keeps gaining even when every source is weaker.
On-policy distillation (OPD) aligns a student model with a teacher's token-level distribution on the student's own rollouts, and it has become a standard way to move capabilities between LLMs. The setup carries a hard precondition: the teacher must be at least as capable as the student. At the frontier that precondition stops holding.
Two common recipes both hit the teacher wall. Distilling a larger model into a smaller one fails when no larger teacher exists above the frontier model you are training. The other recipe (MOPD) trains several domain experts from a shared base and consolidates them, but every expert has to be trained at the student's scale, so cost scales with the student. W2S-OPD (Microsoft Research, University of Maryland, MBZUAI) asks the opposite question: can a stack of models weaker than the student push the student further?
The trick is to build a "proxy teacher" in logit space from a contrast pair: a positive model m+ and a negative m-, both smaller than the student and both cheap. The proxy teacher is:
πT,α(·|st) = softmax(zbase(st) + α(z+(st) − z-(st)))
The z+ minus z- term is the heart of it. Both models are smaller than the student, but their difference isolates the "capability direction": whatever m+ does that m- does not is precisely the shift that capability produces. Add that direction back onto the student's own base logits, and the proxy teacher injects the direction while staying distributionally close to the student, so it does not drag the student off course. α is an amplification coefficient controlling how hard the signal is pushed.
The student then runs standard Top-k OPD against this proxy teacher, minimizing per-token reverse KL on its own rollouts. Compared with vanilla OPD the only extra cost is forwarding three frozen small models (base, positive, negative) for their logits; nothing is trained.
The paper offers three ways to pick the contrast pair, each isolating a different origin of capability:
The student is Qwen3-8B; teachers are only 4B-class. Math is averaged over four benchmarks (AIME24, AIME25, HMMT25 Feb, HMMT25 Nov); code over three (HumanEval+, MBPP+, LiveCodeBench-V6):
| Method | Math avg | Code avg |
| Student base 8B | 17.0 | 57.6 |
| 4B-RL teacher | 48.8 | 61.5 |
| OPD (single teacher) | 46.5 | 58.7 |
| W2S-OPD (single) | 51.8 | 60.9 |
| W2S-OPD (multi) | 52.1 | 61.1 |
The striking line: W2S-OPD hits 51.8 on math and overtakes the 4B-RL teacher it learns from (48.8), while plain OPD (46.5) stays below the teacher. The relative gain over OPD is 11.4% single-teacher and 12.0% multi-teacher on math.
Push the setting to the edge: two untrained base models (4B and 0.6B), both smaller than the student and used without any training, still lift math from 17.0 to 23.0. The correct-vs-wrong-hint contrast, which only needs one 0.6B model and the difference between feeding it a correct or wrong hint, lifts math from 17.0 to 18.4. Every supervision source weaker than the student, and the student still climbs.
Generalization holds out of domain: on GPQA-Diamond, OPD 54.4 to W2S-OPD 56.5; on IFBench, OPD actually degrades the student from 26.3 to 25.9 while W2S-OPD reaches 27.0.
The three contrasts are not redundant. Tagging reasoning tokens with Schoenfeld's eight problem-solving episodes and looking at the top-1% highest-Δ tokens, the post-RL contrast over-weights Plan (15.7% vs 8.6% baseline) and Monitor; the scale contrast over-weights Analyze (44.4%) and Implement (27.7%); the hint contrast over-weights Plan (20.2%) and Answer (16.4% vs 3.8%). Post-RL and hints sharpen the reasoning framework; scale sharpens the solving procedure.
The cost is modest: W2S-OPD adds about 20% per training step over single-4B-teacher OPD, because rollout generation dominates and forwarding three frozen small models is cheap by comparison.
This recasts weak-to-strong from "can the student learn from weak supervision at all" into "how do you extract a capability direction from a pile of weak models and amplify it." The practical payoff sits at the frontier: when the model you are training is already the strongest you have and there is no larger teacher to borrow from, W2S-OPD gives a way to keep gaining using smaller, cheaper weak models. The three contrast pairs cover nearly every source of weak models you would realistically have on hand: pre- and post-RL checkpoints, different-scale bases, even one model fed different prompts.
A dose of skepticism. The dramatic jump (17.0 to 51.8 on math) comes specifically from the post-RL vs pre-RL contrast, which requires you to have run RL on a 4B model to obtain the expert checkpoint, so you did pay for 4B-scale RL. The two settings that use only off-the-shelf untrained models give much smaller absolute gains (23.0 and 18.4). Genuinely free weak supervision helps a little; the big win still leans on a trained expert.
The authors flag open questions themselves: how far weak sources can push a stronger student before saturating, how to elicit more informative signals from them, and whether weak-to-strong becomes a sustained post-training paradigm. None has an answer yet.
The amplification coefficient α is tuned by hand. The paper notes that small α gives weak supervision while large α distorts the distribution and pushes the proxy teacher away from the student; there is no automatic way to set it.
The three contrasts carry very different signals, but the paper gives no rule for picking the best contrast pair for a new task. The multi-teacher route just assigns each query to its most suitable pair, and the routing depends on existing domain labels.
Results are validated only on the Qwen3 family; whether they reproduce on Llama or other families is not shown.