OPSD gains fade to 0.14 at 8B; distilling only verified rollouts keeps a 3-point gain at scale

Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

Md. Ismail Hossain, Humaira Kousar, Isidora Chara Tourni

cs.LG, cs.CL

2026-09-30

OPSD's gains shrink from 3.05 to 0.14 points from 1.7B to 8B. OASIS distills only answer-verified rollouts, holds a ~3-point gain at 8B, and needs final-answer labels only.

What problem this solves

On-policy self-distillation (OPSD) is a cheap recipe for math reasoning: one model plays both roles. The student sees only the problem and samples a solution; the teacher gets the same problem plus a written reference solution and produces a target token distribution at every position of the student's trajectory, which the student matches with a forward KL loss. It uses fewer tokens than RL and avoids the distribution mismatch of offline distillation. Two questions had gone unanswered. It needs a written solution for every training problem, while most verifiable datasets ship only final answers. And nobody had measured which part of the teacher signal actually does the work.

The diagnosis here also explains an awkward fact: OPSD's gain decays with scale. From Qwen3-1.7B to 8B, its improvement over the base model drops from 3.05 to 0.14 points, effectively zero.

Method

The main diagnostic is a recovery probe: cut a student rollout at 10%–90% of its length and let the model continue from the breakpoint under different contexts.

• On rollouts that end in a wrong answer, the teacher holding the gold reference recovers the correct answer in 77% of continuations versus 37% for the student alone. Unrelated or empty references give nothing (32%/37%). The teacher's edge comes entirely from information the student cannot see.
• On rollouts that already end correctly, the gap shrinks to 99% versus 95%. Almost no edge.
• Training logs show only 28.3%/30.6% of rollouts verify at 1.7B/4B, and 62.0%/64.4% hit the token cap without producing an answer. Around 70% of OPSD's updates land on failed trajectories, exactly where the teacher is least imitable. As scale grows the student gets stronger and the teacher's privileged edge falls from +0.40 to +0.17 (1.7B to 4B), but OPSD's supervision distribution never changes.

Proposition 1 formalizes this: since the student cannot condition on the reference, the best it can do is match the context-averaged teacher distribution, which leaves an irreducible loss equal to the conditional mutual information I(A;R|Z), and the average is higher-entropy than the teacher itself. Training logs agree: OPSD's student entropy rises from 0.40 to 0.70 and its top-1 agreement with the teacher falls from 0.932 to 0.888. A separate finding (Proposition 2): the per-entry KL clip in the OPSD implementation (κ=0.05) removes the teacher's upward pull on clipped tokens and reverses the gradient sign. At that threshold it cuts 97%/94% (verified/failed) of the positive KL mass, which plausibly explains the late-training drift both methods show.

OASIS (On-policy Alignment via Scaffold-Isolated Supervision) keeps the OPSD loss, teacher, prompts, temperature, and clip unchanged, and swaps two roles.

• Scaffold (which trajectory to supervise): sample K=8 rollouts per problem, verify final answers, and supervise along the shortest verified one. Problems with no verified rollout are dropped; about half the candidate problems get masked out.
• Context (what the teacher sees): no written solution. The teacher reads another rollout of the same problem, usually the shortest unverified attempt; when all eight rollouts verify, another verified one. The justification is measured: on verified scaffolds the teacher barely responds to its context. Replacing the gold reference with the student's own failed attempt moves the teacher distribution by a total-variation distance of just 0.065.

So the method needs one final answer per problem. Unlike rejection sampling, verification only decides where to distill; the training target is still the teacher's token distribution, not the verified trajectory itself.

Results

Qwen3 at three sizes on AIME 2024, AIME 2025, and HMMT 2025, three-benchmark mean, Avg@12:

• OPSD's gain over base: 3.05, 1.98, 0.14, fading to zero with scale. OASIS: 3.64, 3.84, 3.19, stable at all three sizes.
• OASIS's margin over OPSD grows with scale: +0.59, +1.86, +3.05. The paper concedes the 1.7B margin sits within sampling variation on a 30-problem benchmark and is not conclusive alone; the 4B and 8B gaps are clear.
• GRPO, using the same verifier as a scalar reward, gains only +0.60, +0.60, +0.23, so the dense teacher distribution contributes beyond the verifier. SFT on written solutions never beats base.
• Cost: written-solution baselines train on 12,800 problems; OASIS sees 6,400, trains on about 3,200, and generates 51,200 rollouts, yet matches or wins (63.70 vs 63.60 at 4B; 65.83 vs 64.80 at 8B). It generates about 16x the tokens (43.5M vs 2.76M at 1.7B) while supervising about 30% fewer. K=4 halves generation and costs 1.4 points on AIME24 (52.77 vs 54.17); picking the longest instead of the shortest verified scaffold drops it to 52.33.

Why it matters

The diagnosis stands on its own: OPSD's fading gains are a supervision-allocation problem, not a mystery. The teacher's biggest advantage sits on states the student cannot reproduce, and "distill where the teacher is imitable" applies to any distillation setup with privileged context. Practically, OASIS lowers the data requirement from written solutions to final answers, which opens settings like code with unit tests. The trade is annotation cost for sampling cost, and it pays off where labels are expensive and inference is cheap and parallel.

Keep the frame in mind: every comparison runs at a small budget of 100 steps with LoRA, and GRPO gains only 0.2–0.6 points under that same budget. This is not GRPO's ceiling, and "distillation beats RL" does not follow from this comparison.

Limitations

The authors list three: competition math only, one model family (Qwen3), and single training runs at 4B and 8B, on top of the higher generation cost. Further questions from the read: the headline +3.05 at 8B comes from one training run evaluated once, with only 4B getting three evaluation seeds, so training-side variance is unquantified. The clipping explanation for late drift is labeled by the authors themselves as one possible explanation and is never fixed and tested in isolation. The teacher is the frozen initial model, so the method adds no new knowledge; it reallocates supervision, and where the residual gain at 8B comes from is still worth probing.

Terms

• OPSD (on-policy self-distillation): one model acts as both student and teacher; the teacher additionally sees a reference solution, and the student matches the teacher's token distribution along its own sampled trajectory.
• Scaffold / context: the two roles the paper separates. The scaffold is the trajectory whose prefixes receive supervision; the context is the information the teacher conditions on.
• Privileged information: information available to the teacher during training but not to the student at inference, here the reference solution.
• Imitation gap: the irreducible error left when a learner imitates an expert that observes more than it does.
• Recovery probe: a diagnostic that truncates a rollout and measures how often continuations from the breakpoint reach the correct answer.
• Avg@12: mean accuracy over 12 sampled solutions per problem, lower-variance than a single greedy decode.

Source

• arXiv 2609.37915v1
• PDF
• Hugging Face

Related papers

• Unsupervised partner D rises from 61.3% to 85.5% when JEPA reconstructs only C
• GPT-5.6-sol edge kernels: 64.5% correct, only 25.5% faster than tuned TVM
• Subspace Tuning freezes 99% of each weight matrix's spectrum: forgetting drops from 57 points to ~1
• Stopgrad flow maps converge exponentially even though the update is not a gradient
• Orthogonal 18-harness RL lifts Qwen3-4B in-space Pass@1 by 9.8 over one harness
• Rank-1 Activation Subspace Lifts Qwen3-0.6B Zeroth-Order Tuning Past MeZO

All paper explainers