On-Policy Self-Distillation without Any Supervision
Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, Nuno Vasconcelos
cs.LG
2026-08-07
U-OPSD majority-votes a model's own rollouts into a pseudo-answer and distills it into its wrong reasoning with no labels or rewards, beating supervised OPSD on Qwen3 math by 2-3 points on average.
On-policy distillation and similar post-training methods help large models reason better, but every existing one leans on outside supervision. OPSD feeds ground-truth answers to the teacher as privileged context. GRPO needs verifiable rewards. Some methods depend on a larger teacher model. That works for math or code, where answers are checkable, but fails for tasks with no answer key.
The real question: can a model improve its reasoning using only itself, with no external signal at all? This paper says yes.
U-OPSD replaces external supervision with internal consistency. The whole pipeline runs on the model's own samples:
The design targets exactly where the model is confidently wrong. Multiple samples agree on one answer, so it trusts that answer; yet a few rollouts still go wrong, and those wrong steps are what needs fixing. Dropping the correct path onto the prefix of a wrong one makes the model correct itself at the precise point of failure.
A few engineering choices matter: the teacher uses the shortest correct rollout, since short reasoning tends to be cleaner. The divergence is computed over the full vocabulary by default (57.1); an ablation shows truncating to the top-100 logits is slightly better (59.0), while using only the student's sampled token collapses to 43.5. The objective is forward KL, because reverse KL diverges in training.
Five competition math benchmarks (AIME24, AIME25, HMMT25, MATH500, AMC23), non-thinking mode:
| model | base | +OPSD (supervised) | +U-OPSD (unsupervised) |
| Qwen3-4B | 40.96 | 46.29 | 49.49 |
| Qwen3-8B | 43.57 | 52.04 | 54.31 |
U-OPSD beats the base by 8.5 (4B) and 10.7 (8B) points on average, and beats the label-fed OPSD by 3.2 and 2.3 points. With zero supervision it beats the supervised baseline.
Against other unsupervised methods (TTRL, RENT, Intuitor), U-OPSD leads by about 7 points at 4B and 11 at 8B.
In thinking mode the gains shrink: at 4B U-OPSD scores 77.05, 0.85 above OPSD; at 8B it reaches 77.99, essentially tied with OPSD (77.97). The base is already strong in thinking mode, so majority vote mostly reproduces an already-decent answer.
For tasks with no answer key (open QA, long-form writing, dialogue), this opens a path that needs no reward model, no human labels, and no larger teacher. Wherever majority voting works, U-OPSD works. It turns self-distillation into genuine self-distillation: teacher and student are the same model, and the privileged context is self-generated.
The ceiling is real, though. Majority vote reproduces whatever the base model most often produces, so on problems the base cannot solve, voting will not conjure the right answer.
The authors list several. First, validation covers only the Qwen3 family and competition math; generalization to other models and tasks is unproven. Second, gains depend strongly on mode: large in non-thinking, near-tied in thinking. Third, about 13.3% of pseudo-labels are wrong, so the method is bounded by base competence. Fourth, error bars across random seeds are still pending for every setting.
One open question: the near-zero gain in thinking mode suggests the method degenerates into light fine-tuning once the model is already strong. Whether U-OPSD is therefore more valuable on smaller, weaker models is not broken out separately.