Google paper: auto-optimized agent harnesses overfit evals — RRSI fixes it
omarsar0 · x · 2026-09-24
A new Google paper shows that if you auto-optimize an agent's harness (prompts, control flow, tools, memory), eval scores can go up while the agent gets worse on real tasks.
The problem: automated harness evolution proposes edits, keeps the ones that raise the score, and repeats — effectively overfitting the training tasks. Of five harness-evolution methods compared, Meta-Harness hit 93.0 on the Harvey LAB evolve split but gained only 0.3–1.5 points on JobBench, GDPval, and APEX-Agents.
The fix (RRSI): adds regularization on both sides of the loop — the proposer gets a shrinking edit budget and is pushed toward untried directions, while a critic rejects benchmark-shaped edits.
Result: RRSI scores lowest on the tasks it evolved against but highest on all three out-of-distribution benchmarks, showing how to prevent harness self-evolution from overfitting evals.
More from coding & agent
- Grok Bots 1.0 leaks: multiplayer teammates, proactive agents in Slack and Teams — XFreeze · 2026-11-12
- Could MCP-powered copy-trading disrupt ETFs? Investor floats paying 0.5% for trade signals — MartinGTobias · 2026-11-12
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Ten wild Gemini 4 Carbon demos in 24 hours: CUDA meadows, V8 engines, mech stomps — socialwithaayan · 2026-11-12
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Hamel Husain's complete AI evals framework for PMs and engineers — aakashgupta · 2026-11-12