RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhang, Yoonho Lee, Chengsong Huang, Han Yu, Zhongying CuiZhu, Yifei Ming, Huaxiu Yao, Burak Gokturk, Tomas Pfister, Chen-Yu Lee
cs.LG, cs.AI, cs.CL
2026-09-22
RRSI regularizes harness evolution with an edit budget, leakage screen, and cost gate. Evolve-set gains reach 14.1 points and OOD 4.7, on 2.42M tokens vs 3.80M unregularized.
Most of what an LLM agent can do lives in the harness: system prompts, control flow, tool descriptions, memory, and context trimming. The backbone stays frozen; editing the wrapper changes behavior. A recent line of work lets the model rewrite that wrapper from failed trajectories, which is recursive self-improvement (RSI) at the system layer.
The loop reuses one evolve set to propose, score, and keep winners. Task names, gold answers, and evaluation noise leak into prompts. Evolve-set scores jump; a new tool interface or verifier wipes the gain. Some methods even finish below the starting harness H0 out of distribution.
RRSI leaves the edit space open. Prompts, control flow, tools, memory, and subagents may all change. What it constrains is how the search moves.
On the proposal side:
On the selection side, all of the following must hold:
The backbone is frozen throughout. The main run uses Claude Opus 4.8 as policy, proposer, and leakage critic. Coding starts from Terminus-2; workspace and engineering design use a ReAct loop over an MCP gateway with ReSum-style context management.
Each domain evolves on one suite. The resulting harness is evaluated unchanged on held-out and OOD benchmarks.
Evolve-set gains: +6.0 on Terminal-Bench 2.1, +4.9 on EngDesign, +1.1 on Harvey LAB. With Gemini 3.5 Flash as policy, Terminal-Bench moves from 64.6 to 78.7, which is the "up to 14.1 points" in the abstract.
Out of distribution: SWE-bench Verified +1.8, never scored during search; Harvey LAB in-distribution held-out +2.3; JobBench / GDPval / APEX-Agents +4.7 / +3.5 / +3.7 (relative +7.2% to +13.1%); Frontier-Eng +4.3 Medal points (+24.3% relative). No held-out split regresses.
Against four harness-evolution methods on the same H0 and candidate budget:
| Method | Harvey evolve | JobBench | GDPval | APEX-Agents | OOD avg |
| H0 | 89.4 | 36.0 | 48.8 | 34.2 | 39.7 |
| Meta-Harness | 93.0 | 37.1 | 49.1 | 35.7 | 40.6 |
| TTHE | 91.1 | 35.2 | 47.0 | 31.7 | 38.0 |
| Unregularized | 92.8 | n/a | n/a | n/a | 40.3 |
| RRSI | 90.5 | 40.7 | 52.3 | 37.9 | 43.6 |
RRSI posts the smallest evolve-set gain and the only OOD average more than a point above H0. Drop both regularizer groups and the evolve score hits 92.8 while OOD falls to 40.3 and tokens per trial rise from 2.42M to 3.80M. Drop acceptance only: 91.5 evolve, 41.0 OOD, 3.59M tokens. Drop proposal only: OOD falls 1.7 points.
A coding run with Gemini 3.5 Flash still transfers +2.2 to SWE-bench. The same harness, given to Gemini 3.1 Flash Lite that never joined the search, lifts Terminal-Bench from 11.2 to 14.6. The wrapper is a program, not a set of weights tied to the search policy.
Most agent products already hand-edit harnesses. This paper turns "the wrapper overfits the eval" into a measurable evolve-to-transfer gap, then regularizes search dynamics without shrinking the edit space. If you automate harness evolution, copy the leakage screen and the cost gate first: one blocks answers in prompts, the other blocks score-chasing via longer context.
Engineering-design tasks are graded by frozen simulators, not an LLM judge, and the gains survive, so the method is not just matching a judge's prose.
Only the harness evolves; weights stay frozen. The method still needs a finite evolve set and several hyperparameters, and weak feedback will not be rescued by regularization. Transfer across very different agent architectures, tool stacks, and longer self-improvement runs is untested. Three workspace OOD suites still use a judge, even though the engineering split is deterministic. β0 and β1 are chosen on the evolve set and frozen; there is no sweep on whether they must be re-tuned per domain.
The main policy and the proposer are both Claude Opus 4.8. Gemini transfer works, which does not mean an arbitrary small model can propose edits. The leakage critic is itself an LLM; missed leaks and false kills both steer the search, and the paper does not report critic precision.