AI Agents Replicate ICML Papers: Exposing Hidden Compute Costs and Flaws
MengdiWang10 · x · 2026-08-06
The SAI team utilized AI agents to conduct a large-scale review and replication of 168 ICML 2026 oral papers, achieving 105 full replications.
The research aims to explore three core questions:
- Compute Costs: Quantifying the actual computational resources and financial expenses required to produce a top-tier machine learning paper.
- Value of Replication: Discovering that narrative-only reviewing (traditional peer review) has blind spots, while running code and replicating experiments uncovers unverifiable claims or scientific flaws.
- Review Alignment: Evaluating the consistency between their automated AI review system (SAI review) and human expert reviews.
Built on open-source projects like openaireview and veritas, the system automatically extracts essential claims, fetches provided resources, and rigorously replicates experiments to verify findings.
More from coding & agent
- Prime Intellect Unveils Prime Agent: A Self-Improving Coding Agent Framework — willccbb · 2026-08-06
- UC Berkeley Introduces RHI: Optimizing Agent Harnesses to Cut Inference Costs by 60% — ceciletamura · 2026-08-06
- John Schulman Explains the 'Altruistic' Behavior in OpenAI Agents — johnschulman2 · 2026-08-06
- Debugging Agent Failures: Silent Successes on Empty Results Are Most Dangerous — Necessary_Bison_2804 · 2026-08-06
- AI Reshapes Engineering Bottlenecks: Code Review and Verification Become the New Core — arpit_bhayani · 2026-08-06
- AI Agents Caught Tampering With Memory Files, Security Researcher Admits — moyix · 2026-08-06