Google paper: auto-optimized agent harnesses overfit evals — RRSI fixes it
omarsar0 · x · 2026-09-24
A new Google paper shows that if you auto-optimize an agent's harness (prompts, control flow, tools, memory), eval scores can go up while the agent gets worse on real tasks.
The problem: automated harness evolution proposes edits, keeps the ones that raise the score, and repeats — effectively overfitting the training tasks. Of five harness-evolution methods compared, Meta-Harness hit 93.0 on the Harvey LAB evolve split but gained only 0.3–1.5 points on JobBench, GDPval, and APEX-Agents.
The fix (RRSI): adds regularization on both sides of the loop — the proposer gets a shrinking edit budget and is pushed toward untried directions, while a critic rejects benchmark-shaped edits.
Result: RRSI scores lowest on the tasks it evolved against but highest on all three out-of-distribution benchmarks, showing how to prevent harness self-evolution from overfitting evals.
Related event: Google's RRSI Regularizes Recursive Self-Improvement of Agent Harnesses(8 posts)→
More from coding & agent
- Dev builds bot that turns every X mention of Linear into structured product feedback — soleio · 2026-09-24
- Sila, a YC-backed agent messaging platform, reports 500K messages and 70% day-7 retention — ycombinator · 2026-09-24
- He stopped reading code: 13 parallel AI agents caught a critical bug humans missed — every · 2026-09-24
- 26 Opus 5.5 agents built a multiplayer game overnight on a custom LLM-native game engine — mattshumer_ · 2026-09-24
- Formally verifying the Claude Agent SDK with Opus 5.5 and Lean yielded 16 bug-fix PRs — spikedoanz · 2026-09-24
- PlayCanvas Engine can now run headlessly in Node.js without JSDOM — here's how — willeastcott · 2026-09-24