Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu
cs.AI, cs.CL, cs.IR, cs.LG
2026-10-06
SRD distills trajectory hindsight into pre-action foresight. Added to GRPO or self-distillation, it gains up to 24.2 pp on 10 agent tasks and takes a 2B model from 0% to 60.6%.
RLVR runs usually rely on group-relative objectives like GRPO: sample a group of rollouts for the same task, normalize rewards within the group to get advantages. When every rollout in the group succeeds or every one fails, all advantages are zero and standard implementations drop the group and resample. On long-horizon agentic tasks, where most attempts fail, discarded groups eat most of the sampling budget. Measured across four Qwen3.5 scales, reward-uniform groups account for 37% to 98% of samples, the usable share never exceeds 63%, and the waste is U-shaped: weak models produce all-failure groups, strong models produce all-success groups, and scaling just moves the blind spot from one end to the other.
An all-failure trajectory still contains structured information about how this task kills the agent. GRPO extracts none of it. That is a design gap, not an implementation bug.
SRD (Self-Retrospection Distillation) uses hindsight to supervise foresight. It does not teach the model what to do next; it teaches what the model could have anticipated before acting.
The ablation lands on a counterintuitive default: distill the pitfall channel alone. The knowledge channel's divergence stays flat during training (0.0045, while pitfall falls 30-47%), occasionally hurts, and costs more, since rollouts take 78-87% of each training step.
Ten benchmarks across math, code, search, and agentic tasks, on Qwen3.5-4B/9B-Thinking, avg@8 pass rates:
| Setting | Base | +SRD | Gain |
| 9B OPSD, AIME26 | 52.92 | 77.08 | +24.16 pp |
| 4B GRPO, math avg (3 benchmarks) | 46.22 | 56.25 | +10.03 pp |
| 9B OPSD, math avg | 39.70 | 56.92 | +17.22 pp |
| 9B RLSD, ALFWorld | 66.25 | 77.25 | +11.00 pp |
Gains hold across three base objectives and both scales, with one regression (WebShop under 9B OPSD, −3.61 pp). Training used stdin-format code problems while evaluation used functional-format problems; BrowseComp-Plus runs 10× or more tool calls than training rollouts, and SRD improves five of six settings there, by up to 9.52 pp.
The sharpest number comes from a code-only isolation run: at 2B, 98% of groups are all-failure, GRPO training ends at 0.0% success (1.6% peak), while the same budget with SRD reaches 60.6% training success.
Two side findings. Pure self-distillation at 9B lands below the untrained policy on several benchmarks (HotpotQA −5.50 pp); adding SRD restores the expected ordering and finishes 10.75 pp above it on HotpotQA. Token-level analysis shows SRD does not make the policy talk more (+0.6 tokens per rollout net); it shifts a fixed budget out of inline calculation (−6.7) into naming the problem's structure and plan (+3.7), leaving tool-call syntax untouched.
Reward-uniform groups are wasted compute in every RLVR run. SRD turns them into learning signal at a controlled price: about 10% more wall-clock per step at 4B (197 to 217 s), no change to the base objective, zero inference overhead. Cold-start failures, where the policy is too weak to ever produce reward contrast, are exactly the regime it rescues. Where reward contrast already exists, it works as a consistent positive add-on rather than a replacement.
Stated in the paper: under RLSD, adding the knowledge channel costs 6.88 pp on ALFWorld, and that ablation ran one seed at one scale, so the authors label their explanation a conjecture. Making foresight explicit at inference is nearly inert (one checkpoint gains 5.00 pp, the other three are unchanged within 800-rollout resolution), and a wrong foresight conditions the whole episode; the appendix documents a case. The ethics statement concedes that distilling a model's own hindsight can reinforce its errors and biases.
Discounts from a close read: the main experiments cover only the Qwen3.5 family; the 2B figure of 60.6% is training success, not held-out performance; the hindsight teacher is the same EMA model, so the ceiling of teacher quality goes untested; and at ALFWorld task-type granularity every one of the eight arms trades gains for losses, none improving all six task types.