Recursive Self-Improvement via On-Policy Distillation for Reasoning
Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz
cs.CL
2026-09-25
DCE refreshes the gold-conditioned teacher each round. On Qwen3-8B, Average@12 rises from OPSD's 30.35% to 65.97%; SRCL cuts mean length from 19,046 to 17,561 tokens.
On-policy self-distillation (OPSD) splits one checkpoint into two roles. The student sees only the problem. A privileged copy also sees the gold solution and supplies next-token targets on the student's own prefixes. Freezing that teacher keeps training stable. It also freezes the teacher's revision skill at initialization.
That mismatch is measurable. On fixed wrong Qwen3-8B traces from AIME 2024–2026, the frozen gold-conditioned teacher puts 94.5% probability on EOS at incorrect endpoints, and only 22.0% on the actual reflection cue (often "Wait") in those traces. The student learns to revisit errors. The supervisor still wants to stop. Dense token-level supervision that cannot absorb new revision behavior starts working against the student.
Dynamic Co-Evolution (DCE) refreshes both roles every round. The current checkpoint samples an on-policy solution. The student scores each prefix from the problem alone. The privileged branch scores the same prefix after the gold answer is placed on the assistant side, as if the model had already written it, then stop-gradient is applied. After the update, the new weights initialize both the next student and the next teacher. Revision learned this round becomes gold-conditioned guidance next round.
Where the gold sits matters. Putting it in the user turn costs 9.79, 6.39, and 14.66 Average points at 8B, 4B, and 1.7B.
Stronger revision also rambles. Self-Refined Concise Learning (SRCL) asks the same checkpoint to rewrite its rollout without seeing the gold. A rewrite is kept only if it is shorter, naturally terminated, structurally valid, and answer-correct. Empty acceptance sets drop the SRCL term for that batch. The two losses are weighted together; the main runs use Forward KL.
Training uses the same 14,717 OpenThoughts math problems as OPSD, rank-128 LoRA, learning rate 5×10^{-6}, and non-thinking mode for both branches.
Evaluation is Average@12 on AIME 2024, 2025, 2026 and HMMT February 2025, 12 samples per problem, 32K cap.
| Method | Qwen3-8B | Qwen3-4B | Qwen3-1.7B |
| GRPO | 20.28% | 18.47% | 10.90% |
| OPSD | 30.35% | 22.85% | 10.35% |
| OPSD forced to 16K | 30.76% | 22.99% | 9.24% |
| DCE | 65.76% | 60.00% | 23.33% |
| DCE+SRCL | 65.97% | 61.88% | 26.88% |
On 8B, DCE+SRCL beats OPSD by 35.62 points. Forcing OPSD to spend a full 16K tokens barely moves the needle, so length alone does not explain the gain. SRCL cuts 8B mean length from 19,046 to 17,561 tokens at almost the same accuracy, and lifts 4B from 60.00% to 61.88%. At 1.7B it lengthens outputs from 12,600 to 19,504 while accuracy only goes 23.33% to 26.88%. The brevity objective needs a model that already knows how to revise.
The teacher has to move. Frozen-teacher DCE reaches 47.99% at 8B against 65.76% when both roles update; with SRCL the gap is 41.04% versus 65.97%. On the same wrong traces, teacher endpoint P(EOS) falls from 90.4% to 41.3%, and P(reflection token) rises from 32.8% to 77.4%.
Gemma-4-12B-IT reaches 63.61% with DCE+SRCL against 51.04% for OPSD. Qwen3-14B climbs from 21.39% at init to 66.39% at step 30. Forward KL hits 65.97%; Reverse KL is 5.55 points worse and 6,187 tokens longer; JSD collapses to 17.99%. SRCL without DCE is 2.36% at 8B.
For math post-training without a stronger external teacher, gold is used only to condition the teacher and to filter short rewrites. The dense signal still comes from the model's own rollouts. The copyable knobs are specific: refresh the teacher every round, put gold on the assistant side, distill with Forward KL, and treat SRCL as a length valve, not as the source of accuracy.
This is a post-training recipe, not a new reasoner. The 8B non-thinking 65.97% edges the model's thinking-mode base of 62.08%, still on contest math with LoRA.
There is no dedicated Limitations section. The method needs verifiable gold, so open-ended tasks do not plug in. Each contest set has 30 problems; Average@12 variance will not be small. SRCL lengthens the 1.7B model, which is a warning that concise rewrites ride on revision skill that is already there. The 14B run stops at step 50 and has no matched OPSD baseline. The extra privileged forward pass is not costed out. Scoring "Wait" as the reflection probe is also narrow; real repairs do not always say that word.