Viral "AI beats humans in peer review" study is mostly GPT grading Claude
JessicaHullman · x · 2026-10-25
The widely shared "AI now beats humans in peer review on every measure" study is being toned down by scholars. Jessica Hullman and gburtch note:
- The headline "every measure" comparison is mostly GPT evaluating Claude's reviews — AI judging AI, not human experts
- The more meaningful signal: 3 former journal editors rating human vs Claude reviews, but only for 10 papers, with unclear domain expertise, and they seemingly read the reviews but not the papers themselves
- The authors acknowledge AI is already widely used in peer review, but the conclusions are being overstated in circulation
Lesson: check who the evaluator is before accepting "AI ≥ human" claims.
More from AGI Musings
- Toddler development as a metaphor: evolutionary pre-training and efficient post-training — paigeinsf · 2026-10-25
- Anthropic CEO on learning to code: programming goes first, but your 5% still matters — 2C_ornot2C · 2026-10-25
- AI boom may pose more peril to US markets than meets the eye, NYT analysis warns — nordicinst · 2026-10-25
- After 22 years studying workforces: the AI question isn't replacement, it's 'what kind of people' — 2C_ornot2C · 2026-10-25
- Mechanistic interpretability will never truly be solved, researcher argues — burny_tech · 2026-10-25
- AI-generated physics ideas mostly already done in past 2 months, physicist finds — burny_tech · 2026-10-25