ICML 2026 Reproducibility Audit: Only 7% of Claims Fully Verifiable
机器之心 · wechat · 2026-08-09
SAI recently conducted a massive experimental review of 168 ICML 2026 Oral papers, revealing a severe reproducibility crisis in top-tier AI conferences.
Out of 105 papers fully reproduced, only 34 managed to verify over 40% of their main claims, and a mere 8 papers achieved a reproduction rate above 80%. Common issues included broken code, missing files, and results that failed to match the original text. Furthermore, independent reproduction is highly expensive, with a median cost of $8,900 per paper and the highest nearing $2.2 million.
This high-risk, low-cost academic environment fuels the proliferation of low-quality papers. Ironically, while frontier lab engineers claim they no longer read papers, they still rely heavily on publication metrics to filter talent, placing academia in an awkward and contradictory position.
Related event: ICML Papers Face Reproducibility Crisis in AI Audit(2 posts)→
More from Research
- Star-Forming Region, Not Black Hole, Powers High-Energy Neutrino Factory — AryHHAry · 2026-08-09
- UltraEP: Near-Optimal Load Balancing for Rack-Scale MoE Training — jiqizhixin · 2026-08-09
- LLM Progress Non-linear: Hard Task Gains May Stem from Ceiling Effects — LChoshen · 2026-08-09
- 2026 Dirac Medal Awarded for Physics of Disordered Systems Underpinning AI — S_Conradi · 2026-08-09
- Replacing 32B with 4B: Extreme Compression for MiniMax H3 Video Generation — Fit_Ad7343 · 2026-08-09
- Context Drift in Consecutive Image Edits Remains Unsolved Even in SOTA Models — imdigitalashish · 2026-08-09