AI agents replicated ICML 2026 oral papers, and only 7 mostly held up
ChenhaoTan · x · 2026-07-23
A repost of a research thread about whether science is actually verifiable. The authors replicated ICML 2026 oral papers with AI agents and report that only 27 of 105 papers reproduced more than 40% of the targeted conclusions, while only 7 reproduced more than 80%.
Key findings:
- AI review covered 78% of the comments on which two or more human reviewers agreed.
- For code and reproducibility, the AI reviewers found 903 issues that human reviewers missed, while humans found only 22 issues missed by the AI system.
- Humans still did better on novelty and positioning, suggesting that taste and research judgment remain hard to automate.
The broader argument is that agentic verification may become a useful complement to narrative-only peer review.
Related event: AI Agents Reproduce ICML Papers: Only 7 Out of 168 Fully Hold Up(5 posts)→
More from AGI Musings
- Resy goes scorched earth on agent-based bookings; big agent platforms may force the issue — kevinakwok · 2026-10-27
- Aditya Agarwal: fear is a policy choice, and the biggest risk to American AI — adityaag · 2026-10-27
- The Best University of 2030 Will Have No Professors, Lectures or Exams, Says Predictions Post — nick_linck · 2026-10-27
- Amazon's agent crackdown sparks debate: model personas caught in platform arms race — JeffLadish · 2026-10-27
- Microsoft's reported 60,000 voluntary exits spark essay asking who AI is really for — _jaydeepkarale · 2026-10-27
- Goodharting Isn't Just Fooling Others: How Metrics Confuse Yourself Too — davidmanheim · 2026-10-27