Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3
Jens Lehmann, Sahar Vahdati
cs.AI, cs.SE
2026-10-22
27 lineages built ARC-AGI-3 harnesses from a blank repo and revised them twice, lifting held-out RHAE by only 2.80, with a 95% interval that includes zero.
The program wrapped around a model decides what it sees, what it keeps, which tools it can call, how it recovers, and where the inference budget goes. That program is a harness. ARC-AGI-3 makes the test an interactive grid game: each turn is a 64 by 64 color grid, and the agent is not told what objects mean, what actions do, or what clears a level. RHAE scores only completed levels, applies a square penalty when actions exceed a hidden human baseline, and weights later levels more.
Human-designed systems already reach 100 on the public games. Tycho reports a perfect score with GPT-5.6 Sol at maximum effort and with Opus 5 at xhigh. A thin shell sits much lower. Sol with retained reasoning and compaction scores 38.3 on the public set. Claude Code with Opus 4.8/Fable 5 scores 42.83. Good harnesses are already known to be possible. The question is whether a frontier model can write one itself and then improve it from its own games.
The same model profile does both jobs. The inner loop plays through the task harness. The outer loop edits that code in OpenCode, a fixed coding shell. Weights stay frozen. No person edits the repo or adds a prompt between rounds.
The seed repo has a task note, clients, and prices. It has no solver and no worked game rules. The model must hand over a non-interactive run.sh. The smoke check ignores score: on two development games it must issue a charged action and exit cleanly, with a cap of 50 actions and two levels, and at most ten repairs.
The first accepted version plays five development games fixed in advance: tr87, vc33, r11l, bp35, and ft09. A revision receives frozen source, traces, scores, and dollar cost, and may keep the parent. One further revision follows. The other 20 public games never enter construction, revision, or selection. Nine conditions with three lineages each all finish, for 27 lineages and 81 snapshots.
The adaptation gap is the development gain minus the held-out gain. A positive gap means the feedback games improved by more than the games held out of the update.
Initial held-out RHAE averages 24.54, from 0 to 97.14. One GPT-6 Astra high lineage goes from 97.14 to 97.55 at revision 2. Three Claude Sonnet 5 high lineages finish at 0.11, 0.81, and 0.14. A harness that runs can still barely score.
After two revisions, development RHAE is up 8.42 and held-out RHAE is up 2.80. The 95% bootstrap interval is [-1.44, 7.35] and includes zero. Across 27 lineages, 15 rise and 12 fall; a sign test gives p = 0.70. Inside a descriptive band of 5 points, 15 lineages are indeterminate, 9 rise, and 3 fall. Thirteen held-out games rise on average and seven fall. lp85 gains 20.44 and sb26 loses 15.03. Fully cleared games go from 73/540 (13.5%) to 94/540 (17.4%).
The adaptation gap is 5.62, interval [1.78, 9.77], which excludes zero. Development gains overstate held-out gains.
| Policy | Held-out RHAE | vs initial | USD per game |
| Keep the initial harness | 24.54 | 0 | 10.30 |
| Always deploy revision 2 | 27.34 | +2.80 | 7.57 |
| Pick by development score | 29.06 | +4.53 | 9.15 |
| Held-out oracle | 30.66 | +6.13 | 9.80 |
Choosing the best development snapshot is post hoc. It picks the initial, revision 1, and revision 2 in 9, 6, and 12 lineages, for a held-out gain of 4.53, interval [0.81, 8.68], at $9.15 per game. The rule was not preregistered.
The four score-cost cells hold 9, 6, 9, and 3 lineages: better on both, better score at higher cost, cheaper while losing score, and worse on both. Mean cost falls $2.72 per game. The median lineage moves only $0.54 cheaper. One Sol xhigh lineage goes from 43.95 to 21.99 over two revisions, and from $31.39 to $7.18 per game. One Sol high lineage, in revision 2, raises a self-imposed decision cap from 100 to 120 among other edits; held-out RHAE goes from 29.24 to 53.56, and cost from $8.19 to $8.15.
On Opus 5 xhigh the revision-2 mean is 39.95. Tycho scores 100 on each of the same 20 games. Tycho selected on all 25 public games, so the comparison does not isolate human involvement in the design. The campaign spent at least $18,129.27.
All 27 lineages built a runnable harness from an empty repo and finished both revisions. All 54 revisions read the development evidence, targeted at least one evidence-backed issue, and passed a proximal check. Held-out scores still moved up and down. Broad edits average +3.75 immediate held-out change; edits kept inside the old strategy average -0.20. The interval on that difference includes zero.
In the first revision, 22 of 27 lineages rework memory, replay, when the model is called, or the budget. Four extend the old loop and one mostly preserves it. The second revision splits into 12 reworks, 12 extensions, and 3 preserves. No lineage changes harness family. Median code length grows from 2,161 lines to 2,912.
Keeping the parent was allowed. All 27 scored trajectories changed the code. An updater that cannot stop on weak evidence will keep a lucky development gain. Writing a revision and deciding to adopt it are separate tests.
The study uses one benchmark, five development games, and exactly two revisions. The games are public, so training exposure is possible. Three runs per condition cannot rank models: Opus 5 xhigh moves +8.68 held-out RHAE and Sol xhigh moves -0.05.
Ten rescores of one frozen Sol high final harness give a suite standard deviation of 3.99, and the median per-game standard deviation is 18.78. The main study scores each snapshot once. That lineage stays above its initial 20.38 on every rescore. The pooled +2.80 still has an interval that includes zero, so it is not a settled average gain.
Labels come from two passes of GPT-5.6 Sol xhigh, with no human check. Diagnosis, protection of the old strategy, and resource metareasoning sit below a reliability of 0.67 and are not used to explain outcomes. The other associations are not causal, and score moves together with dollars. Removing development game r11l from the summary drops the gap from 5.62 to 3.02, but the revisions still saw all five games.