AI Agents Reproduce ICML Papers: Over 60% Fail, 99% Have Errors
量子位 · wechat · 2026-08-09
Recent studies reveal that AI agents are being used to systematically audit and reproduce top-tier conference papers, exposing a severe reproducibility crisis in academia.
- Reproduction Failure: An audit firm tested 168 ICML 2026 papers. Of the 92 with verifiable claims, only 34 could have over 40% of their conclusions successfully reproduced by AI, and just 8 reached over 80%. Failures stem from missing code, broken dependencies, or offline models.
- Rampant Errors: A GPT-5-based checker found that 99.2% of top AI papers contain at least one objective error (e.g., math mistakes), with the average number of errors per paper significantly increasing in recent years.
- AI Re-evaluating Science: Beyond catching modern errors, AI has identified flaws in century-old authoritative chemistry data. This marks the first time humanity has the technical capability to systematically review historical scientific literature at scale.
Related event: ICML Papers Face Reproducibility Crisis in AI Audit(2 posts)→
More from Research
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- SERO augments long-horizon agentic RL with executable oracle continuations — Shanghai-AI-Laboratory · 2026-10-16
- Tsinghua and Amap propose OmniVerifier-C1, a graded verifier beating binary judges — Bowei Liu · 2026-10-16
- Alibaba open-sources MAI-Gym, an all-in-one GUI agent eval and RL training platform — Tongyi-MAI · 2026-10-16
- Peking University's SimForcing distills simulation motion priors into robot world models — PekingUniversity · 2026-10-16
- CoWorld Reasoner uses an evolving video world model as a visual reasoning engine — ModalityDance · 2026-10-16