Blog: Measuring Autonomous AI Research with 153 Runs Across 18 Models
eliebakouch · x · 2026-08-16
Prime Intellect published 'Measuring Autonomous AI Research' to evaluate how well frontier models can conduct research. They ran 153 autonomous runs on the nanoGPT optimizer speedrun across 18 models, with individual runs lasting up to eight days. The authors argue that while speedrun methods may not inherently scale to real training, the tight feedback loops make it an interesting testbed. The most striking finding is the significant gap between models, evident in their choice of experiments, execution, and interpretation. No run produced a fundamentally new method, yet Fable 5 and Opus 5 performed dramatically better than the rest.
More from coding & agent
- Dispatch launches newsletter infrastructure built for AI agents, one API call to publish — RichardsonDx · 2026-10-16
- Hyperbrowser launches macOS sandboxes giving agents desktop, apps and Terminal control — madhavsinghal_ · 2026-10-16
- Creator shares the agent-powered system behind millions of monthly impressions — VibeMarketer_ · 2026-10-16
- Devs say LLM coding mistakes are a different class: overcomplication, inferred limits, worse at higher reasoning — Mu5_ · 2026-10-16
- How do you debug bad agent runs when you already have tracing? — Sensitive-Parsnip-12 · 2026-10-16
- Where does pre-task context live for agents? Devs debate workflows beyond handoff — shishen2026 · 2026-10-16