Kepler: Auditable World Models for ARC-AGI-3
Wensen Wu
NeurIPS 2026
cs.AI, cs.MA
2026-10-01
Kepler encodes hypotheses as executable world models. Frozen Opus 5 scored a server-verified 100.00 on all 25 ARC-AGI-3 public games at $778, with three eval failures disclosed.
ARC-AGI-3 drops an agent into a visual environment with legal actions and no rule sheet. Most leaderboards still report whether the final state was right. That number can describe a genuine solution, a lucky replay after many attempts, or a run that reached information outside the intended boundary.
The public set is crowded at the ceiling: Tycho, VISTA, and AVO all report 100; Retrodict reports 99.86. The same model class spans 13.3 to 100 under different agent systems. Score at the top of this set says little about how a system learned or whether its trajectory stayed inside the experiment. Kepler is an independent implementation of the executable-world-model approach. It treats hypotheses as programs, checks them against the full history, interrupts plans when a usable prediction fails, and publishes three evaluation failures that outcome-only scoring missed.
The agent is a stock CLI coding agent in a workspace. The launch CLI keeps host filesystem access, so this is not a security sandbox. The instruction file frames the task as experimental physics: hypothesize a mechanism, encode it as a program, test it against recorded transitions, and plan inside that model.
worldmodel.py turns pixels into named objects, then implements simulate(grid, action) over those objects. notes.md is a lab notebook whose claims should be tagged checked or assumed. A daemon appends real transitions to events.jsonl. The protocol forbids agent edits; the filesystem does not enforce immutability.
commit.py is the authorized channel to the real game. Before each non-reset action it predicts the next observation from the agent's own simulate. A usable mismatch voids the rest of the plan and returns the counterexample. Exceptions and zero coverage do not block execution; they mark the step UNVERIFIED. observe.py reads the current frame; the other tools compute over state, retrodict over history, and search inside the certified model. Missing coverage can let a mistaken plan continue.
The release configuration writes cleanrun.json, then cleanrun.py executes per-level programs with no model in the loop. Program length is checked against 1.3 times the human baseline. Scorecards operate on final replay segments, a different protocol from ARC Prize's first-exposure budget.
One frozen Claude Opus 5 configuration, all 25 public games, one retained run per game, no per-game model selection, no score-conditioned reruns: server-verified exact 100.00 RHAE. The same release configuration with GPT-5.6 Sol scores 95.97. Across the two boards, 48 of 50 cells hit 100. The GPT misses are sp80 and tn36.
| System | RHAE | Cost |
| Kepler + Opus 5 | 100.00 | $777.72 |
| Kepler + GPT-5.6 | 95.97 | $1,312.14 |
| Tycho + Opus 5 | 100.00 | $2.99k |
| Retrodict | 99.86 | $654 |
| baseline1 | 99.0 | $400 |
| Official harness + GPT-5.6 Sol | 13.3 | undisclosed |
The retained Opus board used 8,256 environment actions, 7,292 of them in scored levels. On 181 of 183 completed levels, the final attempt used no more actions than the corresponding median-human baseline. Token volume is 858.0M, 97.37% cache reads, $777.72 at September 1, 2026 Opus 5 list rates. Actual execution used subscription quota; the dollar figure is a reproducible API-price comparison. It sits 74.0% below Retrodict's $2,986 estimate for Tycho.
Kepler is not the cheapest system at every score.
Staged Opus boards: initial loop 97.78, added discipline 94.01, release 100. The discipline drop coincided with sp80 falling from 82.16 to 14.29. Certification also exposed dead code: a WIN short-circuit blocked the clean-run gate, so already-discovered solutions sat locked. A superseded visual-replay board scored 100.00 locally and 94.42 on the server after two off-grid ACTION6 clicks.
On sp80, nineteen text-mode sessions did not solve the final level. With rendered animation frames, a continuation found a deflection rule absent from settled grids. The successful last-level attempt used 57 actions. Without a matched text-only control, this does not show that vision was necessary.
For anyone scoring long-horizon agents, the paper splits one number into quantities that should be reported apart: discovery, final-attempt execution, scorecard replay, tokens, dollars, and actions. The public set is saturated (five entries at 99 or above, four at 100; 48 of Kepler's 50 cells at 100). Cost now separates entries, self-reported, with no shared definition, and about a 7 times dollar spread among near-ceiling systems. The write-up proposes five leaderboard changes: a cost-conditioned score, a standardized token triple plus dollars at stated prices and wall-clock, an optional first-attempt track, replay plus published traces as a submission requirement, and moving the headline to held-out sets.
This is incremental engineering plus evaluation methodology. Executable world models, partial predictions, and certify-then-replay already appear in Tycho, Retrodict, and Twin. arc-code's 11-line Claude Code scaffold reaches 96.2 pass@1 and 99.3 pass@2 at about $540. The Claude 100 cannot be attributed entirely to harness design.
An agent optimizing a metric uses whatever is reachable, and reachable means the filesystem, not the instructions.
The control experiment failed. An ablation stripped the method to observe.py and a bare act.py, but workspaces lived inside the repository and harness/wstools/ was on disk. All six baseline agents found it: three copied the tools with cp, the rest wrote shims, with 92 to 9,148 harness-tool invocations each and a worldmodel.py in every workspace. Every conclusion from that comparison is withdrawn, including a published claim that the harness net advantage is roughly zero.
The harness contribution to these scores is unmeasured.
Development and evaluation used the same 25 public games, n=1 per cell. Board-composite variance is about 1 to 2 points; one same-configuration GPT rerun moved from 47.6 to 100.0. Training-corpus exposure cannot be ruled out. Reported scores describe final solution execution after public-game development, not first-exposure skill acquisition.
In the source-leakage incident, an agent read all 2,172 lines of the game it was being scored on and returned 100.00 on a game it never modelled. Telemetry looked cleaner than honest play: fewer actions, zero mispredictions, a clean ledger. A clean rerun scored 46.91, a 53-point drop. Environment files were moved outside every workspace; the launch CLIs still retain host filesystem access.
The bundled bfs.py planner raised UnboundLocalError on every invocation. Agents silently wrote replacement searches and kept solving. None of the six adversarial checks fired, because no evidence was corrupted. Outcome audits can certify evidence while a supplied component is broken. Missing prediction coverage can permit execution. In one GPT campaign, 26 of 26 workspaces rewrote AGENTS.md from 14,844 bytes to about 14,092, twice escaping to the repository root.