AI Agents Replicate 168 ICML Papers, Only 7 Mostly Hold Up
ChenhaoTan · x · 2026-07-23
A developer utilized AI agents to review and replicate all 168 oral papers from ICML 2026, conducting 105 full replications.
The results reveal that only 7 papers mostly held up. This stark outcome has sparked discussions about a potential "replication crisis" in AI research: whether there is a systemic flaw in the reproducibility of academic papers, or if current AI agents simply lack the capability to effectively execute complex code.
Related event: AI Agents Reproduce ICML Papers: Only 7 Out of 168 Fully Hold Up(5 posts)→
More from coding & agent
- The browser main thread is expensive: a practical guide to JavaScript and CSS animation cost — jh3yy · 2026-09-11
- Inspired by OpenAI's 10,000-agent run, dev open-sources a crowdsourced agent problem-solving platform — Benjaminsen · 2026-09-11
- Lucid: open-source Mac app keeps your laptop awake only while AI agents run — Pitiful_Hedgehog_600 · 2026-09-11
- banteg's snail project crowdsources AI agents to finish matching Snail Mail's 20 remaining functions — banteg · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11
- Kimi K2.8 Preview rolls out: near-K3 coding performance, 1M context for all tiers — teortaxesTex · 2026-09-11