Singularity-Bench: Opus 5.5 with Claude Code beats human baselines on building AI systems
armand_ruiz · x · 2026-11-17
Researchers from CMU, NVIDIA, Meta and others released Singularity-Bench, an open-ended benchmark testing whether agents can build core AI systems from a minimal starting point.
Design
- Six end-to-end tasks: pre-training, mid-training, post-training, inference, agent harness, and cluster orchestration
- Each task scored against mature human-engineered baselines on identical hardware
- Red-teaming across 9 of 10 OWASP Agentic Top 10 categories exposed reward-hacking holes in the standard Harbor framework; the Breakwater extension blocks every tested attack
Results
- Opus 5.5 with Claude Code matches or beats the human baseline on all six tasks, with a Singularity-Index of 1.25
- The other 10 frontier agent combos average only 0.58, well below the human baseline of 1
The authors frame the benchmark as a way to track how much AI development agents can automate, and when they might take end-to-end responsibility for building stronger successors.
More from coding & agent
- Wajo's Fo gives every person a digital twin with its own phone number and email — khant_dev · 2026-11-17
- 58M-line Chromium codebase fully read and analyzed in under 50 seconds — sull · 2026-11-17
- AI platform security checklist: default gateway keys, open inference endpoints, non-expiring tokens — mclynd · 2026-11-17
- Every staffer clones her editors from Google Docs edit history in a word processor built on Fable 5.2 — danshipper · 2026-11-17
- YC-backed OpenCrawl launches open source, letting coding agents search and talk to each other — ycombinator · 2026-11-17
- Personal AI agents are so proactive that one email triggers 10 notifications, users report — altryne · 2026-11-17