What Does Privileged Information Add to On-Policy Self-Distillation?
XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua
cs.CL
2026-09-17
AMPLE-Math holds answers fixed across six reasoning views on 5,319 math problems. On Qwen3-1.7B most OPSD gains are reference-free; a clean solution adds 1.30 points.
On-policy self-distillation freezes a copy of the model as teacher: the teacher sees an answer or a worked solution, the student sees the problem, and the teacher scores the student's own prefixes. A fuller reference looks like more to learn. The open question is how much that adds beyond distillation itself.
Other groups already get gains with no target-specific reference, or even with another problem's text, and structured guidance can beat a full trace. Gains over the base mix the reference with a second asymmetry: a thinking-enabled teacher scoring short direct-response prefixes. Strip the reference and that cross-mode gap remains.
AMPLE-Math takes 5,319 problems from OpenThoughts-114k and writes six answer-matched views: answer only, gist, key points, a polished solution, a narrative summary, and the source full trace. Rendered lengths run from 13 to 4,916 Qwen3 tokens. Splits of 1,536 / 192 / 384 follow how often the frozen base solves each item in direct-response samples.
The main student is Qwen3-1.7B with LoRA r=64, α=128, for 100 steps; SmolLM3-3B is the second family. The teacher is always the frozen thinking-enabled backbone. The primary recipe trains with thinking off and a 1,024-token cap, then evaluates with thinking on. The reference-free control deletes the privileged section but keeps the thinking teacher, so it is cross-mode self-distillation, not matched distributions. The loss is teacher-to-student forward KL with per-token terms clipped at 0.05. Correctness alignment, pressure on reconsideration markers, and the time profile of KL diagnose what the teacher pushes on a given prefix.
On Qwen, the reference-free student already beats the base in domain and on AIME/HMMT, and so does a wrong-answer control that swaps in another problem's answer. At step 100, Answer Only and Full Trace sit about 0.6 points above reference-free, intervals including zero. Clean Solution averages +1.30 across three seeds, interval [0.20, 2.41], and fails Holm correction across six views. The largest lifts land on items the base never solved in four direct-response tries, yet already solves 62% of the time with thinking on.
On SmolLM3, Full Trace adds 2.0 points over reference-free at step 50. By step 100 all three configs fall below the base; Full Trace falls less. Rankings flip under direct-response evaluation: Qwen's reference-free and Answer Only beat Full Trace by 10.81 and 5.32 points, and SmolLM3's step-50 Full Trace goes from +2.00 with thinking on to −10.63 when answering directly.
Training trajectories matter more. Keep the teacher and the reference, switch student rollouts to long thinking-enabled answers, and step-50 gains over the base become losses for all three views, including Qwen's external set and SmolLM3. Replacing a genuine Clean Solution with a length-matched other-problem text costs about two thinking-enabled points. Relaxing loss on sampled wait/but/check markers barely changes marker use and does not raise step-100 accuracy. Extending the loss window from the first 1K tokens to 4K, or sprinkling four 256-token windows, looks like a four-point win only because the baseline stopped at step 50 and the new windows stopped at step 25. At matched steps the edge is under one point, intervals including zero.
People running OPSD should not assume a more complete teacher solution is a better teacher. On Qwen, cross-mode distillation already carries most of the lift; the extra reference effect is small and view-specific. How the student answers, and which inference mode it trains in, steer transfer more than how much of the solution the teacher sees. A privileged reference is worth what it adds to that cross-mode transfer, not how much of the write-up it reveals.
The evidence is short-run LoRA on math in two small families. Only Clean Solution leaves zero in an unadjusted interval, and Holm knocks it down, so this is not a general "references help" result. Thinking-enabled training changes mode, horizon, and the supervised fraction of the response together. Teacher profiles explain the signal; editing the loss often does not move student behavior. External sets have 30 problems each, so intervals are wide.