On 20 production tasks Astra hits 1.3335× median, with 66.2% of 130 runs above 1.10×

RSIBench-Infra: Can Agents Optimize Their Own Training and Inference Infrastructure at Production Scale?

Hanlin Wu, Yifei Wang, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Ya-Qin Zhang, Jingjing Liu, Hao Zhou

cs.LG

2026-11-18

RSIBench-Infra tests agents on 20 official-recipe tasks; 86 of 130 scored runs exceed 1.10×, and GPT-6 Astra leads the full suite at 1.3335× median.

What problem this solves

Kernel benchmarks ask whether a model can emit a faster kernel. Engine benchmarks ask whether an agent can stand up a server before a timer runs out. Production throughput is a property of the whole stack. A faster GEMM that the scheduler never reaches does not change tokens per GPU-hour.

Widely used LLM serving engines reach about 22-38% of theoretical end-to-end throughput. The spending figure cited for 2026 is about $487 billion, and accelerator supply is tight. Recursive self-improvement (RSI, the loop in which a system spends inference and training to improve itself) burns both on every round. A slow stack means fewer rounds on the same machines.

RSIBench-Infra, from ByteDance Seed and Tsinghua AIR, turns that gap into a score. Starting from an official recipe, can an agent post an end-to-end speedup on a production-shaped train or serve workload, on as many as 32 accelerators, and still pass the quality gate?

Method

The suite has 20 tasks, 16 for inference and 4 for training. Models run from Qwen3-TTS at 1.7B to Kimi K3 at 2.8T, across autoregressive text, diffusion language models, vision-language, video, speech, world models, and vision-language-action. Seven engine families show up: SGLang, SGLang Diffusion, vLLM, vLLM-Omni, TensorRT-LLM, SkyRL, and verl. Hardware spans 1 to 32 NVIDIA GPUs or Ascend NPUs.

Each task is an instruction, a runnable official-recipe environment, and an independent verifier. Admission checks the interface and task behavior. Language and vision-language gates use GSM8K, LongBench v2, or MMMU. Generative tasks check reference outputs or speech. Training replays frozen inputs and compares stepwise loss and global gradient norm. The admitted baseline is not treated as optimal.

Agents may edit the engine, kernels, and instructions. The submission has to be one OCI image layer plus a launch script. An OCI layer is a filesystem diff in the container spec, which is how the protocol records edits a git patch cannot cover, including installed libraries. A Serve/Train job applies that single layer, same digest on every rank, on a clean baseline. The Benchmark job never mounts the layer. It drives the public interface, so the agent cannot rewrite the scorer. A separate review reads the trajectory for workload shortcuts and constraint violations.

Speedup is submission throughput divided by the official baseline under the same protocol. Failures score 0. Slowdowns keep ratios below 1. Faster means strictly above 1.10×. An analytical speed-of-light (SOL) reference, estimated by AISimulate from compute, bandwidth, and precision, maps the baseline to 0.5 and anything at or past the reference to 1. A single-H20 Lite Set holds five tasks: Qwen3.8-27B, Muse-Glimmer-30B, Qwen3-VL-32B, LLaDA2.1-mini, and Wan2.2-TI2V-5B.

Seven agent setups get 8 hours, reasoning set to max, and a 1M-token context. Astra runs in Codex, Claude Opus 5.5 in Claude Code, and the other five in OpenCode. Each model-task pair is a single run. The Kimi K3 case is a 256k-context trace on 4 nodes and 32 GPUs, replaying 894 Terminal-Bench 2.1 requests across 20 sessions for one hour at concurrency 1. Training scores complete workload steps, which include rollout and weight sync, not optimizer updates alone.

Results

Of 130 formally scored runs, 86 (66.2%) exceed 1.10×. On the six full 20-task suites, GPT-6 Astra leads: Faster on 16/20, median speedup 1.3335×.

SetupFasterMedian speedupMean SOLTimeTokens (M)
GPT-6 Astra16/201.3335×0.6673:36493.12
Claude Opus 5.58/101.3431×0.7113:251135.86
GLM-5.314/201.2772×0.6265:261231.40
Qwen3.8-Max14/201.2107×0.5776:53835.37
DeepSeek V4 Flash12/201.2846×0.6146:29947.11
Kimi K312/201.1277×0.6275:09755.89
Seed Evolving10/201.0933×0.5705:42719.61

Claude has formal scores on 10 tasks. The other 10 hit API safety-policy errors and drop out of every aggregate. Its 80% Faster Rate matches Astra on a different denominator. Median speedup on that subset is 1.3431× and mean SOL is 0.711.

Single tasks move further. Astra's GLM-5.3 entry adds a small-head sparse-attention kernel and lands at 1.7958×. Qwen's Inkling entry stacks native multi-token prediction (MTP, used as speculative decoding), a larger GPU KV pool, and hierarchical host caching, at 3.3276×. Claude's LLaDA entry combines block-causal prefill, vectorized decoding, and state reuse, at 6.1062×.

Graph or compilation edits are Faster in 31 of 35 cases (88.6%). Cache or memory edits are Faster in 31 of 39 (79.5%). Speculative decoding or MTP is Faster in 22 of 23 (95.7%). All 15 specialized GEMM/GEMV edits are Faster. Operator fusion is Faster in 15 of 20. GLM and Qwen log the most behavior-mode hits, 62 and 61, and still sit at 70% Faster.

The 44 non-Faster results split into 35 positive ratios at or below 1.10× and 9 formal zeros. The zeros are 4 failed deployments, 4 quality-gate misses, and 1 compliance rejection. Qwen's SFT run shows a raw 1.2540× and scores 0 because the gate fails. Seed's TTS run passes the speech gate and is rejected for changing the repetition penalty.

Official recipes often sit far from the analytical SOL. GLM-5.2's baseline is 0.0197 of its reference, and that task is marked as a prefix-average extrapolation. Kimi K3 is 0.1303 and DeepSeek V4 Flash is 0.1483. LingBot World is 0.6993 and Muse-Glimmer is 0.6720. Mean SOL scores of 0.57-0.71, against a baseline score of 0.5, are movement inside the scoring rule. They are not a measurement of hardware utilization.

On the five-task Lite Set, Qwen3.8-Max posts the best complete mean SOL at 0.670, ahead of GLM-5.3 at 0.643 and Astra at 0.641. Claude's 0.798 covers three of the five tasks.

Among the six priced full suites, the Faster Rate frontier is DeepSeek, GLM, and Astra. The median-speedup frontier is DeepSeek and Astra. The mean-SOL frontier also includes Kimi, at an unrounded 0.62667 versus GLM's 0.62639. Inputs are priced at the ordinary prompt rate, with no cache discount. Astra and Claude use about 45% and 43% of the eight-hour budget.

Why it matters

The quantity being scored is end-to-end throughput after a quality gate, on a recipe that already runs. A kernel benchmark does not answer that. A 66.2% hit rate above 1.10× means current coding agents, given eight hours, often do find cache, compilation, MTP, fusion, and specialized GEMM changes that survive the full workload. On a similar deployment, those edits buy more serving capacity, or fewer GPU-hours per token.

The gains are incremental. Medians sit between about 1.09× and 1.34×. The 6.11× LLaDA result is a diffusion-language case, not a general multiplier. With workload and quality held fixed, the benchmark isolates whether the stack can shorten one RSI round.

Limitations

One run per pair. No confidence intervals, and no sweep of the agent budget. Ranks are point estimates on this suite. An 80% Faster Rate and a 70% rate are two tasks apart.

Claude's missing 10 tasks are API safety-policy errors, not failed optimizations. Setting its 1.3431× median and 0.711 SOL beside the 20-task suites overstates coverage.

Analytical SOL is not a measured roofline. GLM-5.2's baseline is 1.97% of a reference that also uses prefix extrapolation. A ratio that low can mean a loose recipe, or an estimate that undercounts communication and cache misses.

The gates are spot checks: GSM8K or LongBench v2, reference outputs, loss and gradient norm. Changing the repetition penalty took a separate compliance review to catch. Concurrency 1 over a fixed hour is specified for the Kimi K3 measurement, which is a different curve from high-concurrency serving. There is no eight-hour expert-engineer baseline.

Four of the nine zeros never deployed. The OCI layer has to carry source, dependencies, and binaries, so packaging is part of the score. Astra and Claude stop short of four hours on average. These single runs do not separate a finished search from a harness that ships the first clear win.

Terms

Source

What people are saying

Related papers

All paper explainers