AI ranked all 6,617 ICML 2026 papers and disagrees with human reviewers
ShayneRedford · x · 2026-10-07
Researchers used AI to rank all 6,617 ICML 2026 papers (dubbed the "AI Awards") and found:
- Humans and AI disagree significantly on paper quality
- AI judgments are strongly influenced by online hype
- AI prefers papers with perceived impact and generalization across settings, while humans favor novelty
- Multiple AI models agree highly with one another
Amid ongoing debate about AI-assisted peer review, this experiment offers concrete evidence that delegating scientific review to AI could systematically shift the direction of science.
More from Research
- GroundedSLAM debuts, decisively beating all methods on Meta's egocentric SLAM benchmark — Scobleizer · 2026-10-07
- HCI researcher begs authors to stop claiming 'reflexive' thematic analysis without reflexivity — IanArawjo · 2026-10-07
- Reza Zadeh claims faster matrix multiplication algorithm, suspects labs near exponent 2 — Reza_Zadeh · 2026-10-07
- Redditor proposes graph-based deterministic modeling to make LLM finance agents trustworthy — jonnylegs · 2026-10-07
- COLM 2026 poster presents scaling test-time compute for agentic coding — dan_fried · 2026-10-07
- COLM 2026 Efficient Reasoning workshop lands Friday, with panel featuring top researchers — tydsh · 2026-10-07