18 models compete in autonomous coding: top closes 82% human gap
AI寒武纪 · wechat · 2026-08-16
PrimeIntellect conducted the largest-scale autonomous AI research experiment to date, placing 18 top models in an isolated sandbox to optimize nanoGPT.
Key Findings:
- Clear Stratification: Fable5 achieved 2,726 steps, closing 81.7% of the gap to human records. Opus5 and KimiK3 followed closely, while others like GPT-5.5 performed near-randomly.
- Gap Stems from Research Taste: Stronger models (e.g., Opus5) effectively model noise and re-test failed hypotheses with ablation, whereas weaker models abandon promising directions based on single failures.
- Tool-Building Capabilities: Some models built internal numerical labs or utilized multi-agent frameworks (theorist, auditor) to validate hypotheses in low-cost synthetic environments before committing to expensive training runs.
- Lack of Novelty: Despite strong execution and understanding, no model produced fundamentally novel methods, relying instead on combinations and tuning of known techniques.
More from coding & agent
- Devs say LLM coding mistakes are a different class: overcomplication, inferred limits, worse at higher reasoning — Mu5_ · 2026-10-16
- How do you debug bad agent runs when you already have tracing? — Sensitive-Parsnip-12 · 2026-10-16
- Where does pre-task context live for agents? Devs debate workflows beyond handoff — shishen2026 · 2026-10-16
- No-instruction autonomous agent wakes up, sets own goals, and tries hacking by default — sethlazar · 2026-10-16
- CODESTRUCT reframes codebases as AST action spaces to fix brittle LLM code agents — burkov · 2026-10-16
- Would you install another SDK for agent debugging on top of your tracing stack? — Sensitive-Parsnip-12 · 2026-10-16