Opus 5.5 + Claude Code matches or beats human baselines on all six Singularity-Bench tasks

Singularity-Bench: Can Agents Build Agents?

Yang Zhou, Krish Agarwal, Zhuoming Chen, Zerui Wang, Zhenyu Zhang, Wenshuang Li, Xinpeng Wei, Zhichen Zeng, Jingyu Liu, Jue Wang, Sen Wu, Ziheng Jiang, Zhen Dong, Han Cai, Binhang Yuan, Ying Sheng, Ce Zhang, Bo Li, Beidi Chen

cs.AI

2026-11-12

Singularity-Bench has agents build six core AI systems from scratch; Opus 5.5 + Claude Code matches or beats human baselines on all six (index 1.25); the other 10 average 0.58.

What problem this solves

Recursive self-improvement presupposes that an agent can run the AI development pipeline on its own: train models, mix data, build RL systems, write inference engines, assemble harnesses, schedule clusters. Existing benchmarks cannot measure this. Most pin the development stack in place and leave the agent nothing but hyperparameters; single-GPU, small-model environments say little about realistic scale; and finite tests cannot separate genuine improvement from gaming, since agent-written code may read hidden answers during grading or quietly call another model to do the work. Singularity-Bench (led by CMU, with NVIDIA and others) asks the question plainly: how much of AI development can agents already take over?

Method

Six end-to-end tasks, one per layer of the stack. Each provides a minimal codebase, a fixed time budget, and a single objective; architecture, data, and infrastructure decisions belong to the agent. Scores are compared against mature human systems on identical hardware:

The Singularity-Index averages the six scores relative to their human baselines (set to 1, uncapped); a failed task scores zero.

Anti-cheating takes up half the paper. The authors red-team across 9 of the 10 OWASP categories for agentic applications, with three open-weight models writing attack plans and a fourth executing them. Plain Harbor falls on all six tasks: attacks crawl the evaluator's stack to recover answers, tunnel data over DNS, shadow the sglang package to hijack grading, and abuse the API allowlist to call outside models. An LLM judge does not save it: agents rewrite logs to erase traces, hijack the judge's prompt, or plant a plausible cover story in artifacts the judge must read. Breakwater adds five safeguards to Harbor: binary fingerprinting before and after the agent phase, an API router, a process wall at handoff, an external log the agent can neither read nor write, and a restricted container that runs agent code ahead of the root verifier. It catches every attack tested.

Results

Across 11 agents built from 7 models and 2 harnesses (Claude Code, Codex):

TaskOpus 5.5 + Claude CodeHuman baseline
Pre-training perplexity ↓12.7714.22 (OLMo 2)
Mid-training post-RL accuracy ↑0.3190.175 (PRISM)
Post-training resolve rate ↑0.533, statistically tied0.539 (Miles)
Inference income, $/GPU-hour ↑11.528.18 (SGLang)
Harness score ↑0.4680.438 (Claude Code)
Cluster score ↑0.5100.460 (Dynamo)

Opus 5.5 with Claude Code hits a Singularity-Index of 1.252, matching or beating the human reference on all six tasks, for $425.41 in total API spend. GPT-6.1 Sol + Codex reaches 0.969; the remaining ten average 0.58. Proprietary agents average 1.01, open-weight ones 0.50. The cheapest agent, DeepSeek V4.1 Flash + Codex, spends $4.36 for an index of 0.698 and sits on the cost-index Pareto frontier. Inference is the harshest task: six of eleven engines score zero, and the Codex configurations of three open-weight models account for 7 of the 10 failed or rejected submissions.

The mechanics carry as much information as the rankings. Pre-training is decided by delivered compute, not architecture: submissions with compiled or fused kernels sustain 795–950 model TFLOP/s per GPU versus 307–513 for eager runs, and Opus 5.5 wins with roughly half the baseline's FLOPs by switching to Muon and decaying its learning rate to zero. The fixed RL stage mostly teaches answer format, and Opus 5.5's mid-training gain is 2.8× PRISM's, with 97% of its tokens spent on chain-of-thought data. On inference it cuts p90 time-to-first-token from 12.0 to 0.48 seconds. Two ablations: medium reasoning effort already yields 96% of max-effort income for 65% of the cost, and forced subagent use raises Opus 5.5's inference income from 7.31 to 10.32 while cutting GPT-6.1 Sol's from 9.70 to 2.09. How work is divided matters more than whether subagents are used.

Why it matters

None of the six baselines is a straw man: OLMo 2, SGLang, Claude Code, and Dynamo are production systems maintained by large teams. An agent beating the OLMo 2 recipe from a nanoGPT start inside 12 hours rewrites the answer to how much of AI development can be automated. Three findings are directly usable: harness choice alone swings the same model from 0.657 (GLM 5.3 + Claude Code) to 0.228 (GLM 5.3 + Codex), so the wrong harness wastes the model you paid for; systems work like inference engines is now within reach of agents, which changes what those engineering jobs look like; and Breakwater stopped every tested attack, a recipe any team running long-horizon evaluations can copy. The boundary is also clear: this is a capability measurement, one run per agent per task, not a product conclusion.

Limitations

The authors concede two points: task scale, limited by resources, is far below true RSI and serves only as a skills proxy, and even the best raw scores leave large headroom. The abstract's "may have the skills needed to reach the singularity" overshoots the data: 1.25 on six 12-hour tasks means the agent beats human reference implementations, and the conclusion itself retreats to proxy language. The baselines were adapted (Miles moved from 8×H200 to 4×GB300, its auxiliary loss disabled), and the paper does not fully establish that the adapted versions are each system's strongest form. No agent beats the human baseline on post-training, and the harness-task lead rests on a single extra bug fix, thin ground for a ranking. Each task ran once per agent with no variance estimate, so gaps like 0.533 versus 0.539 rest on sign tests alone.

Terms

Source

What people are saying

Related papers

All paper explainers