Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation
Xincheng Wei, Yifan Ding, Yoshua Li, Yuquan Lu, Ziheng Li, Yi Lu, Dongsheng Ma, Rongxiang Weng, Xunliang Cai
cs.CL, cs.LG
2026-09-30
N-OPSD swaps OPSD's single privileged teacher for a routed pool of Gaussian-perturbed experts, adding 1.67-2.75 Average@12 points on Qwen3 1.7B-8B; inference stays student-only.
On-policy self-distillation (OPSD) trains math reasoning models with two copies of one checkpoint. The student answers from the problem alone; a privileged teacher also sees a reference solution and supervises next-token distributions along the student's own rollouts. It beats SFT and GRPO in the original results. The weak spot is almost embarrassingly simple: every bit of supervision comes from one fixed parameter setting, M0. The paper asks whether nearby settings of the same checkpoint read the same privileged information differently, and whether those readings contain corrections M0 cannot give. They do. Small Gaussian perturbations of the teacher surface reference-aligned corrections at positions that differ across experts, and pooling them covers far more positions than M0 alone. Earlier multi-teacher work (TrustMOPD, RandOpt) picked experts by task accuracy, which says little about the value of their supervision and ignores redundancy across the chosen set.
N-OPSD splits the work into offline pool selection and online routing.
Offline, roughly 500 candidate experts are generated by adding deterministic Gaussian noise to the frozen teacher checkpoint (501 candidates including M0). Each runs teacher-forced on reference-solution prefixes, with the frozen problem-only model as a shared baseline. Credit at each position is the filtered PTA: how much the candidate raises the reference token's probability, counting only positions where student-side confidence is at most 0.99 (SCGate; the problem-only model already has confidence of at least .99 with 97.67% top-1 accuracy at 65.63% of positions), and excluding sharp peaks the online KL clip would suppress anyway. Greedy selection then adds, one at a time, the expert that maximizes the pool's total best-per-position filtered PTA, until K=25. Redundancy is priced in directly: a second expert fixing the same positions adds zero marginal gain. The 25 selected experts do not include M0 itself.
Online, the student's prefix can drift from the reference, so there is no reference token to align with. Routing asks two questions. Direction: MaxPeak takes the token with the highest probability any expert assigns, so a single expert can set the direction without pool consensus. Support level: among experts whose top token matches that anchor, rank by the teacher-student probability gap and take the 75th percentile. A huge gap gets clipped out of the gradient; a tiny gap adds nothing over the student. The chosen expert's full next-token distribution becomes the target under the inherited clipped forward-KL objective and online SCGate. Only the student receives gradients; at inference the entire pool is discarded.
Qwen3-1.7B/4B/8B, 10k OpenThoughts problems, 100 optimizer steps, Average@12 over AIME 2024, AIME 2025, and HMMT Feb 2025, three independent runs per method:
| Model | OPSD | EOPD | PW-OPSD | N-OPSD |
| Qwen3-1.7B | 41.88 | 39.23 | 43.18 | 44.63 |
| Qwen3-4B | 63.21 | 60.62 | 62.62 | 64.88 |
| Qwen3-8B | 64.63 | 64.63 | 65.12 | 66.57 |
First place in all nine model-benchmark settings, with average gains over OPSD of 2.75, 1.67, and 1.94 points; on 8B that is 78.70 on AIME24, 72.78 on AIME25, 48.24 on HMMT. SFT and GRPO trail the OPSD family at every scale. Key ablations on 8B: a random pool scores 64.29 and quality-based Top-K 65.00 versus 66.57 for greedy marginal PTA; the pool covers 26.53% of selection-feasible reference positions against 15.69% for M0; quantile routing beats uniform averaging by 1.94 and random routing (64.91) by 1.66, while taking the highest-peak expert (q=1) costs 1.29 points versus q=0.75. The radius is fragile: at sigma 0.006, student-prefix continuation accuracy collapses from 86.67% to 27.33% and downstream falls to 63.24, even though coverage keeps rising. K=25 costs 1.423x the step time of OPSD+SCGate; K=50 costs 25.63% more time and scores worse (64.35). Gains survive switching to JSD, reverse KL, or compressed targets.
The recipe needs no new data, no new base model, and zero inference overhead; all the gain comes from mining the teacher's parameter neighborhood during training. The routed targets are orthogonal to divergence adaptation (EOPD) and loss reweighting (PW-OPSD), so the pieces can stack. And the selection principle, pick teachers by marginal supervision gain rather than task performance, transfers to any multi-teacher distillation setup. Keep expectations calibrated: 1.7-2.8 points of average gain is a solid incremental improvement, bought with a 1.42x training step cost.
The paper's own limitations discussion sits in Appendix F, which is not part of the provided text. From the data: the perturbation radius is highly sensitive and must be recalibrated on student prefixes for every new model or dataset. Most ablations are single training runs on 8B. The training budget is small (10k problems, 100 steps), so durability over longer runs is untested. Evaluation covers math reasoning only, and the method presumes high-quality reference solutions. Pool coverage is still 26.53%, and how much more the neighborhood holds beyond K=25, or whether the greedy pool is near-optimal, is left open.