AI Ranks All 6,617 ICML Papers as Area Chair — and Disagrees Sharply With Humans
ShayneRedford · x · 2026-10-07
Shayne Redford and Seungone Kim had LLMs act as area chairs, ranking all 6,617 ICML 2026 accepted papers anonymously. Key findings: AI and humans disagree significantly (Kendall's τ ≈ 0.08); AI is heavily swayed by online hype; AI favors perceived impact and generalization while humans prefer novelty; different AI models agree strongly with each other. The authors stress they aren't advocating AI ACs — they want to expose the systematic biases AI already introduces into science.
More from AGI Musings
- Pilot study: AI agents rank all ICML 2026 papers as area chairs, and their 'research taste' differs from humans — ShayneRedford · 2026-10-07
- Neuroscientists debate whether brain properties left out of computational models matter for consciousness — eschwitz · 2026-10-07
- 1,200 people reproduce 2,226 ICML papers with agents; AI reviewers diverge from humans — evijit · 2026-10-07
- Mathematician Daniel Litt: the math profession will change enormously, but nobody knows how — littmath · 2026-10-07
- 'The only moat left is caring about your project past two days' — danshipper · 2026-10-07
- Gary Marcus Asks: Is a 'Nice Tool' Worth a 10% Risk of Catastrophe? — GaryMarcus · 2026-10-07