AI as Area Chair: Model Ranks All 6617 ICML Papers, Diverges Sharply From Humans
Shayne Redford and Seungone Kim's team ran a pilot study called "AI as Area Chair": language models played the role of area chairs, anonymously ranking all 6617 accepted papers at ICML 2026 without seeing authors or human review scores, and the results were compared against human ACs' oral/spotlight decisions to probe how AI's "research taste" differs from that of human researchers.
Confirmed
- The AI's top ten is dominated by reinforcement learning papers, led by "Spurious Rewards: Rethinking Training Signals in RLVR," with self-distillation reinforcement learning research also included
- The divergence is striking: only about a quarter of human-selected orals made it into the AI's top 10%; across all main-track papers, agreement between the AI ranking and the oral/spotlight tiers and review scores (Kendall's τ) was only about 0.08
- The authors clarified their stance: they are not advocating for AI as ACs; AI is already deeply involved in brainstorming, writing, reviewing, and revising research, and the study aims to make the consequences visible, including the risk of introducing uniform, systematic bias into science
Why it matters
As AI agents increasingly take part in the research process, models with consistent, systematic "tastes" influencing topic selection, reviewing, and writing could create homogenizing bias in academia. This study is the first to quantify such divergence at the scale of a full top-conference submission pool, providing a data foundation for discussing the risks of AI involvement in peer review
2026-10-07 ~ 2026-10-07 · 6 related posts
Primary sources
- AI Ranks All 6,617 ICML Papers as Area Chair — and Disagrees Sharply With Humans — ShayneRedford ·
- Only a Quarter of Human-Picked ICML Orals Reached the AI's Top 10% — ShayneRedford ·
- AI-as-AC Authors: Not Advocating AI Reviewers, But Warning of Systematic Bias — ShayneRedford ·
- [source] AI Ranks All 6,617 ICML Papers as Area Chair — and Disagrees Sharply With Humans — ShayneRedford · 2026-10-07
- AI Area Chair's Top 10 ICML Papers: RLVR Signals, Self-Distillation RL Lead — ShayneRedford · 2026-10-07
- [source] Only a Quarter of Human-Picked ICML Orals Reached the AI's Top 10% — ShayneRedford · 2026-10-07
- [source] AI-as-AC Authors: Not Advocating AI Reviewers, But Warning of Systematic Bias — ShayneRedford · 2026-10-07
- Pilot study: AI agents rank all ICML 2026 papers as area chairs, and their 'research taste' differs from humans — ShayneRedford · 2026-10-07
- AI ranked all 6,617 ICML 2026 papers and disagrees with human reviewers — ShayneRedford · 2026-10-07