CMU's Singularity-Bench: Opus 5.5 builds AI infra beating SGLang, scores 1.25 vs human baseline 1.0
新智元 · wechat · 2026-11-16
A new CMU paper introduces Singularity-Bench, six end-to-end agent tasks covering the full AI infra stack: pretraining, mid-training, post-training, inference engines, agent harnesses, and cluster orchestration — work that normally takes elite engineering teams months.
- Of 11 agents tested, only Opus 5.5 + Claude Code met or beat human baselines on all six tasks, scoring 1.25 on the Singularity-Index (human baseline = 1.0); the other 10 averaged 0.58, with GPT-6.1 Sol at 0.969
- Its self-built inference engine yields $11.52 per GPU-hour vs SGLang's $8.18 — on a 100k-GPU cluster that's $8M extra per day, effectively 40% more throughput for free
- It outperformed NVIDIA Dynamo on cluster orchestration; only post-training ended in a near tie (-0.6%)
- The authors argue this suggests frontier models may already possess skills needed for recursive self-improvement
More from Infra
- WSJ ran 66 Nvidia/Jensen headlines in 4 days, joke 'benchmark' shows — xiaosun86 · 2026-11-16
- Inkling too small to saturate Tensor Cores; BBBK speedups scale with model size — andrew_n_carr · 2026-11-16
- SemiAnalysis: renting US compute to China yields Nvidia ~$7M profit per megawatt — xiaosun86 · 2026-11-16
- Reuters: Nvidia chips pile up in Hong Kong as China withholds import approval — _weiping · 2026-11-16
- Tongxin Intelligent raises hundreds of millions RMB to build AGI-native inference chips — 量子位 · 2026-11-16
- GLM-5.3-Flash on two Mac Studios hits ~140 tokens/s via tensor parallelism — cryotic · 2026-11-16