Shockingly Simple Self-retrospection Improves Agentic Models Without RL
Jonathan Light, Christopher Zhang Cui, Jeonghye Kim, Roger Creus Castanyer, Emiliano Penaloza, Zhengyan Shi, Alessandro Sordoni, Marc-Alexandre Côté, Xingdi Yuan, Minseon Kim
cs.AI, cs.CL
2026-09-29
ROFT fine-tunes Qwen3.5-4B on self-retrospections only. It hits 49.2%/26.8% on SWE-bench Verified/Pro after 20 updates, versus GRPO's 48.0%/25.3% after 40, with no RL or teacher.
Most agent post-training still runs RLVR (reinforcement learning with verifiable rewards): a test pass or fail scores the whole trajectory, and methods such as GRPO turn relative success inside a group into an advantage. One binary outcome does not say which assumption failed. An all-failure group has no within-group contrast, so the policy-gradient term is zero. That gap is acute in long-horizon software engineering, where many tool calls sit between the issue and the test.
Prior work already uses reflection, but as a different object. Reflexion keeps the text in the next prompt. STaR, CFT, and SCFT train on critiques or on successful revisions those critiques produce. The open question is narrower: if the model is trained only to explain its own completed attempt, with actions masked and no reflection left in context at test time, does later solving improve.
Microsoft Research, with RPI, UC San Diego, KAIST, and Mila, isolate that transfer with ROFT (Retrospection-Only Fine-Tuning). A retrospection is a post-hoc writeup of the attempt: what happened, which decision mattered, and when to apply the lesson.
The loop is:
Masking actions is the experimental point. If the run also cloned the trajectory, a score bump could come from imitation rather than from explanation. The retrospection may discuss actions and fixes; the supervised object is still the after-the-fact account. There is no teacher, no reward-weighted policy update, and no filter on explanation quality. Failed and successful attempts both count. Retrospection inputs are truncated by field, with the trajectory summary capped at 24,000 characters. Training uses AdamW at 1e-6, 64 source attempts and up to 256 retrospections per update, on four training plus four inference NVIDIA B200 GPUs.
The base Qwen3.5-4B scores 44.2% on SWE-bench Verified and 23.9% on Pro. Training data is SWE-rebench-767, kept only if the base model mixed success and failure across three tries (the learning zone). That filter helps GRPO, which needs reward variation inside a group. Across all 500 Verified tasks with 16 samples, 10.4% are always solved, 61.8% mixed, 27.8% never solved. Tests are held-out Verified and the harder, longer-horizon SWE-bench Pro.
| Method | SWE-bench Verified | SWE-bench Pro | Budget |
| Qwen3.5-4B base | 44.2% | 23.9% | none |
| ERL | 45.6% | not reported | not reported |
| CFT (gpt-6-luna teacher) | 46.0% | not reported | not reported |
| SCFT | 46.8% | not reported | not reported |
| SSD | 47.0% | not reported | not reported |
| GRPO (40 updates) | 48.0% | 25.3% | 8.30 h, 11,576 attempts |
| ROFT (20 updates) | 49.2% | 26.8% | 3.05 h |
The GRPO baseline already includes DPPO-Binary-TV, length shaping on successful traces, and DAPO-style dynamic sampling. ROFT reaches 49.0% Verified after ten updates and 1.29 hours, above that GRPO run at 40 updates. Relative to the 40-update GRPO checkpoint, 20 ROFT updates use 63% less wall time. GRPO's training reward keeps rising through 90 updates while Verified falls to 46.0%. The paper's guess for ROFT flattening near 20 updates is repetitive retrospections on the same tasks, which add less new supervision.
On single tasks where 64 base samples all failed, 40 ROFT updates produce 1.75% online solve rate on a SymPy issue and 3.29% on a SymbiFlow issue. A near-frontier Pylint task moves from 2/64 to a smoothed 12.00%. GRPO gets no advantage from an all-failure group. ROFT still gets a training target from the failed-attempt writeup.
After ten updates, correct recorded turns increase in likelihood 32.41% of the time under ROFT versus 21.21% for incorrect turns. GRPO barely separates the two (44.14% vs 45.30%). After one ROFT update, next-token shifts on frozen traces land mostly in reasoning: task-balanced forward KL is 3.45 times that of tool calls, and larger on early turns and Read/Glob than on Write/Edit. A prompt that asks how to solve more directly, still with a retrospection-only loss and no length penalty, cuts later assistant tokens by 11.7% and turns by 13.1%. On Qwen3.5-9B, 20 updates give 58.8% Verified for ROFT versus 55.8% for GRPO, in 1.93 versus 5.33 hours.
Dropping the final pass/fail verdict leaves Verified at 49.2%. Online generation scores 49.2%, fully offline 48.0%, frozen retrospection generator 47.4%. Four retrospections per rollout beat 2 and 8 in the sweep. Doubling weight on evidence, corrections, or tokens that attend strongly to the user prompt lifts solve rate on 64 training tasks from 50.63% (Uniform) to 54.22% to 54.69%. Those figures are not held-out.
For teams training coding agents, this is a cheaper early-training recipe than the enhanced GRPO they compared against. Same eight B200s: ten ROFT updates hit 49.0% Verified in 1.29 hours, against 8.30 hours for GRPO at 40 updates. The method does not need a final verifier label. Failures are usable. The retrospection prompt is a behavior knob: ask for more direct solutions, and later traces shorten without a length penalty.
A workable mechanism is that predicting retrospection tokens backpropagates through attention into masked trajectory representations, so shared parameters move and later actions follow. ROFT is a minimal isolation study, not a bid to retire RL. Gains arrive early and flatten near twenty updates. Language about an attempt is a training target in its own right, sitting beside action imitation and scalar rewards.
The authors flag cost. Long-horizon agent runs are expensive, so the study stays on software engineering and mostly Qwen3.5-4B. One 40-update GRPO job is about $500 of GPU time; one SWE-bench Pro eval is about $200. They chose simplicity over an optimized recipe. Self-written retrospections can lock in wrong explanations. Turn correctness labels in the behavioral analysis come from GPT-6 Astra, not human audit.
The paper also disagrees with itself on Pro: the abstract and Figure 1 report 26.8%, Section 3.1 and Figure 4b report 29.0% at the same 20-update checkpoint. Main tables do not show multi-seed error. The weighting study is measured on 64 training tasks. The GRPO baseline stacks 2026-era extras; the gap versus textbook GRPO is unreported. Asynchronous generation changes batch composition, so exact reruns will drift.
The authors' own next problem is the right one: decide which explanations deserve to enter the weights, so agents do not internalize fluent but false writeups.