Prime Intellect's Largest Autonomous AI Research Experiment: 18 Models, 153 Runs, Fable 5 Closes 82% Human Gap

Prime Intellect published the blog post "Measuring Autonomous AI Research," detailing the largest open experiment on autonomous AI research to date: 18 frontier models performed 153 autonomous runs on a nanoGPT optimization task, each up to 8 days in 8xH200 sandboxes. Fable 5 ranked first with 2,726 steps, closing 81.7% of the gap to the human record, with Opus 5 and Kimi K3 close behind; all results and traces were open-sourced.

Confirmed

• Setup: The task is the nanoGPT optimizer speedrun track in 8xH200 sandboxes. Per @新智元, sandboxes use bwrap plus network namespaces for offline isolation, each run gets a dedicated 8xH200 node, and models run fully autonomously for up to 8 days without architecture changes.
• Scale: 153 autonomous runs across 18 frontier models, including GPT-5.6, Claude, DeepSeek V4, and Kimi K3.
• Leaderboard: Fable 5 first with 2,726 steps, closing 81.7% of the performance gap; Opus 5 and Kimi K3 follow. Per @AI寒武纪, the 18 models show clear capability tiers with Fable 5 in the top tier.
• Human baseline: The best run closed the gap against a record set by dozens of humans over several months.
• Open data: Full evaluation results and per-run traces are open-sourced.
• Scale comparison: @arohan calls it the largest public experiment of autonomous agents iterating in a research environment, extending runtime, compute, model diversity, and harness types, and compares against tasks in OpenAI/Anthropic system cards (e.g., Anthropic's LLM training optimization on CPUs).
• Multi-agent framework: Per @量子位, Prime Intellect's PrimeIntellect Harness delegates monitoring and implementation tasks to low-cost models, substantially cutting the cost of "AI developing AI" in the nanoGPT experiment.

Unconfirmed

• Post m15 mentions Codex (GPT 5.5) and Claude Code (Opus 4.7) autonomously iterating 10,000 runs and beating the human baseline; its model versions and run count (10,000) conflict with the primary posts' GPT-5.6/Opus 5 and 153 runs. Several reposts (m2–m5, m7, m10, m13) cite "10+ models, 100+ runs," likely rounding; whether they refer to the same experiment is unclear.

Why it matters

• One of the largest open autonomous-research evaluations, fully open-sourced and reproducible, enabling horizontal comparison of 18 frontier models.
• The best run closed over 80% of the gap to the human baseline, directly testing agents' ability to iterate autonomously over long horizons in real compute sandboxes.
• PrimeIntellect Harness demonstrates a low-cost engineering path for "AI developing AI."
• Reddit user adssidhu86 called it a good start for AI self-improvement, but cautioned on whether signals generalize from a single benchmark.

2026-08-16 ~ 2026-08-18 · 18 related posts

• 18-Model AI Research Experiment: Fable 5 Closes 82% of Human Gap — eliebakouch · 2026-08-16
• [source] Blog: Measuring Autonomous AI Research with 153 Runs Across 18 Models — eliebakouch · 2026-08-16
• [source] Leaderboard: Fable 5 Tops Autonomous Research, Kimi K3 Follows — eliebakouch · 2026-08-16
• Frontier Model AI Research Experiment: Best Runs Close 82% Gap to Human Records — eliebakouch · 2026-08-16
• Autonomous Agents Beat Human Baseline in nanoGPT Speedrun Experiment — eliebakouch · 2026-08-16
• [source] NanoGPT Speedrun Frontier: 153 Autonomous Runs Across 18 Models Open-Sourced — samsja19 · 2026-08-16
• Largest Open Experiment on Autonomous Agents Compares Optimizers — _arohan_ · 2026-08-16
• 18 models compete in autonomous coding: top closes 82% human gap — AI寒武纪 · 2026-08-16
• AI Self-Improvement: NanoGPT Speedrun Frontier Experiment Insights and Limits — adssidhu86 · 2026-08-16
• PrimeIntellect releases full traces from largest open experiment on autonomous AI research — burny_tech · 2026-08-17
• Experiment: How Frontier Models Perform AI Research Autonomously — eliebakouch · 2026-08-17
• DeepSeek V4 Pro Chains of Thought Fun; AI Research Experiment Closes 82% Gap — burny_tech · 2026-08-17
• 18 AI Models Ran Autonomous Research for 8 Days; Best Closed 82% of Gap to Human Record — 新智元 · 2026-08-17
• Open Source Infrastructure Beats GPT-4 in Research Tasks via High-Throughput Multi-Agent Framework — 量子位 · 2026-08-17
• Largest open experiment yet: autonomous agents iterate on research tasks for 24h+ — eliebakouch · 2026-08-18

3 near-duplicate retellings: eliebakouch · inductionheads · burny_tech