Shadow Evaluations: AI Agents' Open-Ended Research Rejected by Original Authors
A recent preprint introduces a testing framework called 'Shadow Evaluations' to examine whether frontier AI agents can handle open-ended scientific research. In the experiment, agents took on core open problems from two unpublished NeurIPS papers, with 6 days and thousands of dollars in API and compute resources for autonomous exploration. Ultimately, the original paper authors reviewed the AI-generated research papers and rejected them all. The study highlights clear bottlenecks for agents in open-ended research, providing an important counterweight to overly optimistic narratives of 'automated AI science'.
Confirmed
- The team proposed the 'Shadow Evaluations' framework, where AI agents autonomously explore core open problems from two unpublished NeurIPS submissions.
- Agents were given 6 days and thousands of dollars in compute and API credits, running hundreds of experiments.
- Original authors rigorously reviewed the AI-generated papers and rejected all of them.
- Agents performed reasonably on engineering/coding tasks with easily verifiable results, but struggled in open-ended research requiring hypothesis generation and evidence evaluation.
- The study identified five recurring failure modes of agents in open-ended tasks.
Unconfirmed
- The authors note that the negative results are preliminary, with limitations in sample size and potential for scaffolding improvements; agents' research potential remains to be further evaluated.
Why it matters
- Current evaluations of AI agents often focus on narrow, easily verifiable tasks, whereas real scientific research is open-ended with unclear goals. This study directly addresses this gap, revealing significant bottlenecks in deep innovation tasks and providing a valuable benchmark framework for assessing agents' true research capabilities.
2026-07-30 ~ 2026-08-01 · 12 related posts
- Episode 1: Shadow Evaluations: AI Agents' Open-Ended Research Rejected by Original Authors(2026-07-30, 12 posts)
- Episode 2: AI Agents Lack Metacognition, Act Like Students Doing Homework(2026-07-30, 2 posts)
- Episode 3: AI Agent Autonomous Research Fails: Papers Rejected(2026-08-01, 2 posts)
- Frontier AI Agents Can Do Coding, But Fail at Open-Ended AI Research — Peter Kirgis · 2026-07-30
- AI Agents Fail Open-Ended Research: Papers Rejected by Original Authors — sayashk · 2026-07-31
- Agents Struggle with Open-Ended AI Research: Preprint Identifies 5 Failure Modes — random_walker · 2026-07-31
- Frontier AI Agents Fail Open-Ended Research, Papers Rejected by Authors — sayashk · 2026-07-31
- AI Agents Fail Open-Ended Research: Authors Reject Papers Generated with 6 Days & Thousands in Compute — sayashk · 2026-07-31
- Can AI Agents Conduct Open-Ended Scientific Research? — ChenhaoTan · 2026-08-01
- AI Agents Fail at Open-Ended Research: Automation Remains Out of Reach — sudoraohacker · 2026-08-01
5 near-duplicate retellings: sayashk · sayashk · sayashk · sayashk · RexDouglass