Study: LLMs rewriting their own ARC-AGI-3 harnesss gain just 2.8 points, not significant
omarsar0 · x · 2026-10-25
A controlled study on ARC-AGI-3 asked whether frontier models can build and improve their own agent harness. Each model wrote a complete harness from a fixed coding bootstrap, played five development games, then rewrote its entire codebase twice; every version was scored on 20 held-out games using Relative Human Action Efficiency (RHAE).
Key findings:
- Across 27 lineages, first harnesses ranged from zero to near-ceiling, but two full rewrites lifted mean held-out RHAE by only 2.80 points (95% CI [-1.44, 7.35])
- Development-game gains overstated held-out improvement, and revisions that cut inference cost often hurt task performance
- Human-steered systems still score higher on the same games, so better designs exist — models just can't find them reliably on their own
The authors argue this exposes a gap between autonomous harness construction and reliable self-adaptation, central to what general intelligence requires.
More from coding & agent
- One prompt turns NYC weather data into a perfect 1990s-style academic web page — vboykis · 2026-10-25
- SaaStr founder: AI agents now pick and churn your SaaS vendors — win the agent — morganb · 2026-10-25
- Dev compares Codex vs Claude: one needs constant steering, the other gets intent — menhguin · 2026-10-25
- After reading 1,000+ papers: 25 real-world AI agent use cases worth building — MaryamMiradi · 2026-10-25
- AI trip planner redesign: code review catches accessibility regression in nav links — SeanOliver · 2026-10-25
- Anthropic guide: fold unnecessary subagents into skills and tools, with 2x faster execution reported — omarsar0 · 2026-10-25