Self-Evolving Agents through Privileged Retrospective Distillation
Yuhang Zhou, Yifan Wu, Bo Peng, Mingyi Wang, Jiayi Liu, Xiangjun Fan, Lizhu Zhang, Zhuokai Zhao
cs.LG, cs.AI, cs.CL
2026-10-20
RSD distills hindsight re-scores into the policy. Qwen3-14B ScienceWorld success rises from 12.9 to 20.6 (+59.7% relative); frontier context edits add up to 44.7% relative.
Multi-turn logs pile up after deployment, and the weights usually stay put. Changing prompts, workflows, or memory changes behavior while the skill sits in the scaffold. Changing weights usually needs successful traces for supervised fine-tuning, or environment rewards for reinforcement learning. The setup here is closer to deployment: one trajectory from the current policy per task, no gold action sequence, and no reliable reward. Keeping only successes throws away failure feedback the interaction already paid for.
At action time the policy sees only the history. After the episode, later observations show what earlier decisions caused, and the experience bank holds how related tasks were solved or failed. RSD turns that after-the-fact gap into dense supervision for the current policy.
One model is both student and teacher. The student looks only at the history it had and samples an action. The teacher does not write a new trajectory. It reads a privileged retrospective note, available only after the episode, and re-scores actions the student already took. The student pulls its next-token distribution toward the teacher with forward KL. The loss has no policy gradient and no task reward, so failures still supervise.
The note is rebuilt every round from a bank that accumulates both outcomes. Precedents are filtered related traces: a completion signal on household and science environments, answers grounded in the top retrieved passages on SearchQA, then the top 2 by content-token overlap with the current goal. Lessons are written by the same model, once per task type per round, from successes and failures together. Reflection, once the episode is over, names the earliest pivotal mistake and a better action. Future hindsight appends the later interaction and environment feedback.
Training runs for 10 rounds of 200 tasks, full-parameter updates, learning rate 3×10⁻⁶, forward KL on the top 20 teacher tokens, one epoch per round. The teacher follows the student until the selection metric falls below its peak, then stays at that checkpoint. The main scores are the evolved student with the note removed.
A stronger loop uses Claude Opus 4.8 to edit how the note is built. The whitelist is precedent count and retrieval, component switches, injection position, and teacher temperature. The training algorithm is not editable. Each candidate gets a no-training probe, greedy decoding on a small slice of the training distribution, then one full round. The rule is kept only if environment success is no more than 0.5 percentage points below the best accepted configuration. Otherwise the rule reverts and the new weights stay. The environment metric does not enter the policy loss.
The main table averages three seeds. ALFWorld reports the mean success rate of the seen and unseen splits, ScienceWorld reports development-set success, and SearchQA reports exact match.
| Method | Metric | Result |
| Base → RSD | ALFWorld-8B SR | 62.9 → 76.6 |
| GRPO / GT-OPSD | ALFWorld-8B SR | 75.5 / 70.1 |
| Base → RSD | ALFWorld-14B SR | 69.3 → 79.6 |
| GRPO / GT-OPSD | ALFWorld-14B SR | 82.6 / 82.5 |
| Base → RSD | ScienceWorld-8B SR | 15.6 → 23.4 |
| BoN-16+SFT | ScienceWorld-8B SR | 20.8 |
| Base → RSD | ScienceWorld-14B SR | 12.9 → 20.6 |
| Base → RSD | SearchQA-14B EM | 35.4 → 38.7 |
| Harness@Inf | SearchQA-14B EM | 41.4 |
The largest gain against the base model is ScienceWorld-14B, 12.9 to 20.6, 59.7% relative. On ALFWorld-8B, RSD at 76.6 edges GRPO at 75.5. GRPO uses environment rewards and 8 samples per task; RSD uses one sample per task. On ALFWorld-14B, GRPO at 82.6 and GT-OPSD, which feeds the teacher a gold trajectory, at 82.5 still beat RSD at 79.6. ScienceWorld has no GRPO number. GT-OPSD falls to 3.0, and BoN with 16 candidates plus success filtering reaches only 20.8 and 20.4.
SearchQA stays in a narrow band across all 10 rounds. On the 14B model, Harness@Inf feeds the privileged note to the base model at inference and does not train, scoring 41.4 exact match against RSD at 38.7.
The 8B ablation shows the pieces swap importance by task. On ALFWorld, removing lessons, reflection, or future hindsight drops success from 76.6 to 71.4-71.6, while removing precedents only reaches 75.0. On ScienceWorld, removing precedents drops 23.4 to 15.1, near the base 15.6, and removing future hindsight drops it to 17.6. Lessons from successes only, or failures only, score 69.4 and 72.6 on ALFWorld, below 76.6 when both are used.
At round 3, ALFWorld-8B unseen success is 67.9 with a full refresh, 69.0 with lessons only, 63.8 with precedents only, and 59.7 if the round-1 context stays frozen.
Internalization compares the evolved model without the note to the base model with the note. ALFWorld is ahead by 14.5 and 8.9 points at 8B and 14B. ScienceWorld-8B is ahead by 3.8. At 14B the two are about even, and putting the note back on the evolved model adds 6.1. SearchQA shows no internalization. Actions the teacher disfavors lose probability in the next round early on: from round 1 to 2, Δlog p is −0.384 on ALFWorld and −0.696 on ScienceWorld, then near 0.
After the frontier model edits the context rule, ScienceWorld moves from 23.4/20.6 to 27.5/29.8, a further 44.7% relative at 14B against the fixed rule. Random search on the same budget scores 23.2 and 25.6, so at 14B it already beats the fixed rule and still trails the frontier. ALFWorld only moves from 76.6/79.6 to 78.4/80.3. Without the probe, ScienceWorld-8B falls from 27.5 back to 23.9. All four peak configs increased the precedent count.
With logs on hand, no gold traces, and no reward inside the loss, RSD writes the interaction into the weights. On household and science tasks the success rate stays above the base model after the retrospective note is removed at test time, and above the route that samples more traces and fine-tunes only on successes.
The improvement is incremental and stops at the task boundary. Search question answering barely moves over 10 rounds, and an external note at test time is more useful. When rewards or gold traces are available, GRPO and GT-OPSD still rank higher on ALFWorld-14B. The extra 44.7% relative also calls Claude Opus 4.8 and uses an environment metric to keep or drop context rules.
The policy loss really has no reward term. Precedent filtering still checks whether the environment marked the task complete, and lessons are still written from successes and failures separately. Building the note without success labels is untested.
If the teacher is replaced by the latest student every round, a single ALFWorld-8B seed reaches 80.3 at round 7, 39.4 at round 9, and 0 at round 10. Freezing at the peak avoids that collapse. The teacher schedule and the lesson-source ablation are single-seed. The main table uses three seeds.
Gold supervision can lose to the base model. GT-SFT on ALFWorld scores 34.1 and 42.7, against base scores of 62.9 and 69.3. No cause is given.
Reflexion does not update weights. It spends extra attempts. Three attempts beat single-attempt RSD in several cells: 24.1 versus 20.6 on ScienceWorld-14B, and 43.6 versus 38.7 on SearchQA-14B.
Precedent count flips sign across environments. With 4 precedents, ALFWorld-8B reaches 78.8 and ScienceWorld falls to 20.3. SearchQA is not part of the frontier search.