AI Agents as Researchers: 6 Days, $1000, Engineering Pass but Research Fail
RexDouglass · x · 2026-08-01
A new empirical paper investigates whether AI agents can autonomously conduct open-ended scientific research. The researchers introduced a 'shadow evaluation' method: frontier AI agents were given the core open-ended questions of two high-quality unpublished NeurIPS 2026 papers, along with six days and thousands of dollars in compute.
The results show that while agents completed all engineering implementation without human help, they made no substantial progress toward answering the core research questions, leading to unambiguous rejections by the original authors.
The paper identifies five recurring failure modes: poor judgment regarding the bar for publishable research, lack of creativity, over-reliance on known methods, inability to debug complex issues effectively, and a lack of deep insight into experimental results.
More from AGI Musings
- Miles Cranmer's talk: how to avoid 'AI Brain Rot' and intelligence atrophy — Scobleizer · 2026-10-16
- AI's fundamental design flaws are fully hidden by its products' superficial design flaws — lescarr · 2026-10-16
- IMF Official: Markets' Surprising Resilience Largely Rests on AI Optimism — kernelangus420 · 2026-10-16
- Can AI trained only on human content infer the shape of the human brain? — rickasaurus · 2026-10-16
- Vals RSI Index extrapolates Google models hitting 130% autonomous AI R&D by Oct 2027 — HaydnBelfield · 2026-10-16
- Gowers' tl;dr: no p(doom), but can't confidently say it's close to zero — michael_nielsen · 2026-10-16