1,200 people reproduce 2,226 ICML papers with agents; AI reviewers diverge from humans
evijit · x · 2026-10-07
A Hugging Face hackathon had 1,200+ community members use coding agents to reproduce ICML 2026 papers claim by claim — 6,816 Trackio logbooks covering 2,226 papers, a third of the conference, in 19 days. With 23,918 submissions and 6,352 accepts (roughly double, partly agent-driven), volunteer reviewing hasn't kept up; one spotlight reviewer admitted not checking the proofs, which HF then did check. Separately, an 'AI Awards' experiment ranked all 6,617 ICML papers with AI, finding humans and AI disagree significantly, AI is swayed by online hype, AI favors perceived impact and generalization while humans prefer novelty, and AIs agree highly with each other.
More from AGI Musings
- Video: The Metaphysical Impossibility of AI, on the Boundaries of Machine Minds — ReallyNotARussianSpy · 2026-10-07
- Superforecasters + AI only narrowly beat AI alone at forecasting — random_walker · 2026-10-07
- Matt Shumer's AI-era observation: ambitious friends work more, 9-5 friends coast — mattshumer_ · 2026-10-07
- A Claude-Built Model Estimates How Big a UBI Robots and AI Could Fund Without Inflation — Competitive_Travel16 · 2026-10-07
- Grady Booch: LLMs Are Single-Column Systems vs Brain's 2-4M Cortical Columns — Grady_Booch · 2026-10-07
- 100,000-Person AI March Organized via Assurance Contract Could Be Largest Ever — DKokotajlo · 2026-10-07