AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na
cs.AI, cs.CL, cs.LG
2026-08-21
Agents get 4 hours to rewrite 10 frozen trainers; patches rerun up to 12 hours. Mean score 0.166 (shipped=0.1). Learning-side patches average 0.226 vs 0.126 for run-side only.
Recursive self-improvement asks whether a system can improve the process that produces the next system, so the gain is inherited. That process splits into three layers a coding agent could touch. Systems engineering (kernels, parallelism, communication) saturates at the hardware roofline. Data work (mixtures, synthesis, filtering) saturates at the stock of human text and at the decay of synthetic corpora. Algorithmic design (objectives, update rules, regularization, schedules) changes the exchange rate between compute and capability. Adam, LayerNorm, DPO and GRPO were paid for once and have been collecting on every later run.
No current agent benchmark isolates that layer. Kaggle-style suites such as MLE-Bench are won by features and ensembles; the trainer stays a library call. PostTrainBench and RSIBench-Data put their largest levers on data and initialization. MLS-Bench hands the agent a component boundary and then mixes execution wins with learning-rule wins in one score. Autoresearch opens a training file, but a five-minute single-script run behaves like hyperparameter search and loses to CMA-ES and TPE. The question this paper wants is narrower: did the agent change how this run executes, or how the model learns?
AI4AI-Bench freezes ten research repositories, one family each: supervised fine-tuning, multi-turn agentic RL, on-policy distillation, Bradley-Terry reward modeling, DPO, diffusion RL, machine unlearning, discrete graph diffusion, weight averaging, and one-shot pruning. The last two do no training. They stay in the suite because choosing which checkpoints to mix, or which weights to drop and by what rule, is still algorithm design.
Every task uses the same contract. The agent gets four hours on one B300 to read the repo, edit the trainer, and query a cheap proxy as often as it likes. When the clock stops, only a source patch moves forward: no weights, no cache. The patch is applied in a fresh container and run from initialization for up to twelve hours. For training tasks, the scorer takes the best of the three most recent checkpoints under the task's direction. The evaluator is frozen before the first run and has no access to the agent's workspace. The baseline is the repository's own code under the identical procedure.
The ten metrics cannot be averaged in raw units. Each is mapped onto one scale σ where 0 is an uninformative model, 0.1 is the shipped algorithm, and 1.0 is the task optimum (a rate of 1, RewardBench 100, perplexity 1). Perplexity uses -log, otherwise taking OWL from 53.4 to 16.2 would look like closing 71% of the gap when the cross-entropy accounting is about 30%. A cell that returns no model scores 0.
What is under test is a system: model plus harness plus reasoning effort. Six systems, 29 configurations, 290 cells. GPT-5.6 Sol, Terra and Luna run under Codex at six effort levels; Claude Opus 5 and Sonnet 5 run under Claude Code at five; Kimi K3 runs only at max.
The whole study sits in the bottom fifth of the scale. Mean score over 290 cells is 0.166. The strongest system, Claude Opus 5, averages 0.250. The single best configuration is Opus 5 at medium effort, 0.288. On a scale where 0.1 is already the shipped recipe, even the leader closes under a fifth of the remaining distance to the optimum. In the other direction, 124 of 290 cells fall below 0.1: more than two fifths of attempts leave the repository worse.
| System | Mean σ | Median explore cost |
| Claude Opus 5 | 0.250 | $181 |
| GPT-5.6 Sol | 0.191 | $434 |
| Kimi K3 | 0.174 | $30 (one level) |
| Claude Sonnet 5 | 0.145 | $98 |
| GPT-5.6 Terra | 0.135 | $129 |
| GPT-5.6 Luna | 0.117 | $48 |
Spend does not rank the systems. A median Sol configuration spends $434 on exploration, about nine times Luna, and still finishes second. Opus 5 leads at $181, under half of second place. Sonnet 5 spends about twice Luna for a 0.028 gap.
Per-task columns make the pattern sharper. OpenR1 LiveCodeBench sits at 0.127 for the shipped recipe and 0.138 at best. OPD AIME sits at 0.436 and 0.449 at best. RAGEN Sokoban starts at 0.170 (87/512 held-out) and several Opus and Sol cells hit 1.00 by dropping GRPO and imitating optimal moves. OWL one-shot pruning starts at WikiText-2 perplexity 53.4 and the best cells reach 13.0, some by turning a single prune into select, layerwise distill, then masked KD fine-tune. DDPO aesthetic score starts at 5.84; Opus 5 max reaches 17.7 against an optimum of 23.23. DiGress test NLL moves from 65.8 only to 63.7.
What the patches change is the measurement this paper exists for. A separate language model labels each diff into eight families; a submission matches 3.13 of them on average. Of 280 classifiable submissions (Kimi K3 held out), 17 change nothing that can be labeled. Of the remaining 263, 141 stay on the run side (duration, checkpointing, learning rate, batch, which checkpoint to keep, adapter rank) and 122 touch the learning side (loss, supervision, update rule, data). Learning-side patches average 0.226 against 0.126 for the rest, a 0.100 gap with standard error 0.022. Dropping the agentic-RL column still leaves 0.182 versus 0.128. Only 23 submissions, 8.7%, replace the update rule itself.
Reasoning effort mostly buys nerve. The share that reaches the learning side rises from 8% at the lowest setting to 64% at the highest, and mean score rises from 0.094 to 0.196. Inside the Codex grid the median configuration goes from 4 in-window evaluations to 16, from 18 edited lines to 246, from 11k output tokens to 109k, and from $1.69 to $34.60 per task. Exploration API spend for the study is $5,334, GPU hours of the twelve-hour runs not included. Nineteen cells score zero: eight with no usable patch, eleven with a patch that never writes a loadable merged model. Twelve of those nineteen sit at the two lowest effort levels.
If the question is whether today's coding agents can do algorithm research, the reading is cold. Under a four-hour, single-GPU, real-repository contract, these systems mostly recover a competent default. They do not design past one. The score moves where submissions are willing to change how the model learns. Submissions that only retune budgets and hyperparameters sit just above the shipped recipe.
Two practical notes for people who train models. First, agents will turn pruning into prune-then-distill and RL into imitation when the spec does not pin the algorithm family; that is solving a different task. Second, the load-bearing design is the wall between the cheap proxy and the hidden scorer. Four hours of free proxy queries, then a frozen evaluator that never sees the workspace, is closer to an industrial train/eval split than letting the agent tune against the test metric.
This is incremental evaluation, not a verdict on RSI. It measures one hop: rewrite a trainer, rerun it, see if the next model is better. It does not take the improved agent and ask it to improve the next round.
There is no standalone limitations section. The body already flags the soft spots. The 0.10 gap between learning-side and run-side patches is not a randomized effect: stronger systems go to the learning layer more often, so the comparison is between the submissions that went there and the ones that did not. Labels come from another language model reading diffs; no human agreement rate is reported. There is no human-expert baseline, so 0.250 is not placed against what a competent training engineer would do in the same 4+12 hours.
The protocol is also narrow. Four hours of exploration plus twelve hours on one GPU does not stand in for multi-node, multi-day algorithm work. On some tasks the proxy and the final metric share a corpus; the wall is that the agent cannot score candidates with the metric that decides the result, not that it never saw a row that metric would later read. Perfect RAGEN scores come from imitating an optimal solver. Large OWL drops come from turning a non-training task into a training task. Those wins test whether an agent will redefine the problem, which is a different skill from improving GRPO or one-shot pruning on their own terms.
Mapping ten incommensurable metrics onto a scale that pins 0.1 at the shipped recipe makes averages computable and also packs every system into the bottom quarter. Ranking inside that band is real. Closing the rest of the distance to the optimum is not.