ICML 2026 Reproducibility Audit: Only 7% of Claims Fully Verifiable
机器之心 · wechat · 2026-08-09
SAI recently conducted a massive experimental review of 168 ICML 2026 Oral papers, revealing a severe reproducibility crisis in top-tier AI conferences.
Out of 105 papers fully reproduced, only 34 managed to verify over 40% of their main claims, and a mere 8 papers achieved a reproduction rate above 80%. Common issues included broken code, missing files, and results that failed to match the original text. Furthermore, independent reproduction is highly expensive, with a median cost of $8,900 per paper and the highest nearing $2.2 million.
This high-risk, low-cost academic environment fuels the proliferation of low-quality papers. Ironically, while frontier lab engineers claim they no longer read papers, they still rely heavily on publication metrics to filter talent, placing academia in an awkward and contradictory position.
Related event: ICML Papers Face Reproducibility Crisis in AI Audit(2 posts)→
More from Research
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- DIY LEGO bench tests AI brick-building, GPT-6 Astra beats Opus — TheLegoless · 2026-10-16
- tszzl's theory: Tornado → Apache license → apaches → Southwest → Colorado — tszzl · 2026-10-16
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- A Qwen3 neuron tracks both Colorado and Tornado HTTP code; tszzl offers a theory — tszzl · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16