Agent Plasticity: Measuring Self-Improvement Through Experience
Harman Singh, Anton Bakhtin, Rulin Shao, Gabriel Synnaeve, Ilia Kulikov, Rob Fergus, Sanjeev Arora, Kurt Keutzer, Jason Weston, Anuj Mahajan, Anirudh Goyal
cs.AI
2026-10-07
Weights frozen; experience persists only as tools and memory. Sol's board-game plasticity is 298 pts/$1k; Fable 5 scores higher, with plasticity 57. OOD transfer is partial.
Most agent evals ask what a fixed set of weights can do right now. Self-improvement asks what happens after the same failures and the same number of revision rounds. One model turns that experience into tools the next instance can call. Another revises for many rounds and still sits near its starting score.
A final score hides the split. A strong start ranks well without learning. Similar gains can cost very different amounts. A gain inside the training games need not survive a stronger opponent. The measurement therefore tracks three things together: whether held-out play improves beyond the games that produced the learning, how much that gain costs, and where the loop breaks. Weights stay frozen. What carries across games is only scripts, skills, and notes the model writes itself. The work comes from Meta Superintelligence Labs, UC Berkeley, Princeton, and the University of Washington.
At each checkpoint the agent is a frozen model plus a persistent inventory of executable scripts, natural-language strategies, and text memory. Every game starts from an empty context. Chat history and temp files do not carry over, so a higher score cannot hide in the transcript. Training games supply trajectories, outcomes, errors, and move-quality feedback. The same model proposes the next inventory. A revision is kept only after structural and executable checks. Rejected attempts leave the inventory unchanged and still count toward cost.
Evaluation has three splits. Only training trajectories enter reflection. Held-out in-distribution games match the training difficulty and test whether the artifacts help on games reflection never saw. Held-out out-of-distribution games use stronger opponents and test transfer across difficulty. Board-game score is 100 times (wins plus half the draws), divided by the number of games. Timeouts, missing moves, and unfinished games count as losses.
Plasticity is held-out gain divided by learning cost, in score points per $1,000. Cost includes tokens for acting, reflection, and rejected revisions, converted to dollars. Plasticity to saturation uses a monotone fit and the first observed checkpoint where the fit reaches 90% of its rise, then divides that gain by cost at that point. If the 90% rise is under 10 points, the value is 0. If no checkpoint reaches it, extrapolation stops at three times the cost already spent.
Failures fall into three groups: no artifact covered the mistake, a covering artifact went unused, or it was used and the mistake remained. After checkpoint 0, about 98% of chess moves match at least one relevant artifact. Labels combine a semantic judgment with execution traces. The analysis is observational.
Chess runs for 20 updates from an empty harness. The hard setting plays 16 training games, 12 in-distribution games, and 12 out-of-distribution games per round, against Stockfish limited to 20,000 nodes per move. Go is 5 by 5 and Hex is 6 by 6. NetHack uses the coding agent's own harness, checkpoints 0 through 40, ten fresh games each round.
On the combined board-game curve, hard and easy chess, Go, and Hex are averaged with equal weight. Plasticity to saturation differs by an order of magnitude: GPT-5.6 Sol at 298 points per $1,000, Claude Opus 5 at 233, Claude Fable 5 at 57, Claude Opus 4.8 at 14, and GPT-5.6 Luna at 0. GPT-5.5 is still climbing slowly, so saturation is not identified. Fable 5 has the highest fitted endpoint. In hard chess, the in-distribution score moves from 37.5% to 73.3% averaged over checkpoints 16 to 20 for Fable 5, from 25.0% to 66.4% for Opus 5, and from near 0% to 36.9% for Sol. Sol's absolute score stays lower. It also starts below Opus 4.8, then passes that model on both held-out score and plasticity. Opus 4.8's visible gain is mostly in the easy setting, from 0% to 19.0%.
Gains against stronger opponents are usually smaller. In Go, Fable 5 goes from 20% to 80% in-distribution and from 0% to 50% out-of-distribution. In Hex, Sol goes from 40% to 77.5% and from 28.1% to 78.1%. The ranking flips with the task. Opus 4.8 loses ground in Go and improves in Hex. Opus 5 rises in Go, then a tool-building error on the last round knocks the score down.
In NetHack the clear gain belongs to Claude Opus 5.5, measured as a Claude Code agent. Mean score moves from about 2019 over checkpoints 0 to 2 to about 5831 over checkpoints 36 to 40. The fit reaches 90% of its rise at checkpoint 11, after $1,072.73, for a normalized plasticity of 61.75 points per $1,000. The other five lines show no reliable rise. Sol, the most efficient model on board games, falls from about 570 to about 420. At checkpoint 10, on the same ten games, Opus 5.5 averages 9015 with 2,713 lines of tool code; Opus 5 writes 8,237 lines and averages 1919. More code does not buy the higher score. Opus 5.5 already led by 2.6x to 25x at checkpoint 0, and its three best games supply 69% of the checkpoint-10 total.
One Fable 5 lineage wrote its own chess engine: a search patch after leaving the king in check, a repetition fix after drawing won positions, and extra search time in endgames. Every later game inherits those edits.
Low reuse barely moves the score. Pooled across hard and easy chess, the share of relevant decisions with no artifact use is about 98% for Luna, 87% for Gemini 3.1 Pro, 51% for Opus 4.6, and 28% for GPT-5.5, against 2% to 6% for Fable 5, Opus 5, Opus 4.8, and Sol. Those four reuse artifacts on 94% to 98% of relevant decisions and still post very different scores. They make about 0.16 identified failures per relevant decision, versus about 0.38 for the others, and 83% to 99% of the remaining failures happen while an artifact is in use. Reuse can also move ahead of the score, or against it. By checkpoint 3, Sol and Opus 4.8 already reuse on 93% to 99% of hard-chess decisions, while their in-distribution peaks land at checkpoints 12 and 5. In Go, Opus 4.8 raises successful reuse to 93% as its in-distribution score falls from 40% to the 10% to 25% range.
| Setting | Comparison | Result |
| Board-game plasticity to saturation (pts per $1,000) | Sol / Opus 5 / Fable 5 / Opus 4.8 / Luna | 298 / 233 / 57 / 14 / 0 |
| Hard chess, in-distribution, checkpoints 0 to 16-20 | Fable 5 / Opus 5 / Sol | 37.5% to 73.3% / 25.0% to 66.4% / near 0% to 36.9% |
| Go, Fable 5; Hex, Sol | In-distribution / out-of-distribution | 20% to 80%, 0% to 50%; 40% to 77.5%, 28.1% to 78.1% |
| NetHack, Opus 5.5 | Mean score; plasticity | about 2019 to 5831; 61.75 |
Endpoint skill and acquisition efficiency are different columns. For agents that keep editing tools after deployment, a single static leaderboard promotes models that start strong and do not absorb feedback. On board games Sol buys the most held-out score per dollar. Fable 5 reaches the highest fitted endpoint, at about one fifth of Sol's plasticity.
Low-plasticity models tend to leave relevant artifacts unused, so retrieval and invocation are the nearer fix. High reuse with failures still on the board points to artifact quality, generality, or application. The same interface separates models in all four tasks, and the order is not stable: NetHack and the board games can reverse it. The runs measure plasticity. They do not train for it.
Environment feedback is not separated from the compute spent reflecting. There is no matched-budget control that revises artifacts without seeing the games. Plasticity depends on provider prices, score headroom, horizon, and the saturation fit. Small gaps are a weak basis for a universal ranking. GPT-5.5 never establishes saturation in these runs. NetHack scores are normalized with fixed anchors, 213.6 to 0 and 6526.98 to 100, so they are not on the chess win-rate scale. Opus 5.5's rise also sits on a large head start.
The failure labels are observational. Semantic judgments come from Gemini 3.1 Pro, which is also one of the models under test. Lineages are trained separately, so nothing here shows that a chess engine written into the inventory would transfer to another game. Checkpoint noise remains, and the contribution of long-context instruction following is not isolated.