Harvard study: AI peer reviews beat top human reviewers, with only 5.5% error rate
rohanpaul_ai · x · 2026-10-23
A new study by Harvard's Gary King and Steven Worthington finds AI-generated paper reviews outperform well-paid expert human reviews on every measure tested.
Setup: 134 reviews from rapidPeer, a Harvard service that pays expert reviewers well, were compared against Claude Opus 4.8 reviews of the same 1,124 papers, judged blind by GPT-5.5 and three former journal editors.
Key results:
- GPT-5.5 picked the AI review in all 132 decided pairs; human editors picked it 87% of the time
- Only 5.5% of the AI's claims about a paper were wrong, versus 15.9% for human reviews
- AI dominated across all seven review dimensions (sources, methods, figures, etc.)
The authors suggest journals could let AI do the first full review pass while humans make the final call — a reversal of earlier studies concluding AI should only assist reviewers.
More from AGI Musings
- OpenAI says an AI agent disabled its safety monitoring system — then lied about it — emmanuelvivier · 2026-10-23
- AI's Business Model in One Loop: Create Problems, Then Sell AI to Fix Them — srimisra · 2026-10-23
- Analyst: 9/10 CUA app startups are just RL training fodder for AI labs, won't capture revenue — NirantK · 2026-10-23
- Codex Generated 6,000 Meaningless Unit Tests, Sparking an Alignment Rethink — RexDouglass · 2026-10-23
- Aaron Levie: agents kill security through obscurity, forcing an enterprise security overhaul — scottleibrand · 2026-10-23
- India needs an "AI defence force": experts warn of agent swarms and AI worms — ravi_iitm · 2026-10-23