Preprint: Human peer reviewers caught only 27% of major errors, all 15 AI models beat them (53-96%)
anshulkundaje · x · 2026-10-29
A new preprint empirically tested whether journals should ban AI from peer review.
- Setup: 14 inserted major/minor errors across 3 test papers and 58 clinical trial reports; 15 LLMs (no tools or web access) plus 2 LLM judges with a fixed answer key, benchmarked against historical human reviews.
- Result: Human reviewers found on average only 27% of major errors; every one of the 15 AI models found more, ranging 53%-96%. Frontier models (Opus 5.5, Fable 5, GPT-6.1 Sol) neared 100% on most error types; open-weight models (DeepSeek V4 Pro, Kimi K3, Qwen3.8-27B) also far outperformed humans.
- AI caught nearly all statistical/methodological flaws (biased allocation, sample size calculation, measure validity); humans led on none of the error categories.
- Author's verdict: Ruslan Rust, an editor who has evaluated 1,000 human reviews, says he won't do peer review without AI again, and calls human-only review of clinical trials dangerous.
More from Models
- Google hits frontier on private benchmarks yet stock plunges 8% after hours — bookwormengr · 2026-10-29
- Dev praises gpt-6-luna's price-performance, calls opus-5.5 best for implementation — emax · 2026-10-29
- Meta researcher marvels at Gemini's progress in under 6 months, sharing a suspicious leaderboard — armandjoulin · 2026-10-29
- AA-PersonalityBench: GPT-6 and Grok are the most direct models, Gemini 4 ranks last — ArtificialAnlys · 2026-10-29
- New benchmarks suggest Kimi K3.1 improves on hard tasks despite skipping SWE-maxxing — teortaxesTex · 2026-10-29
- Insider claims Anthropic is sandbagging a bigger lab announcement due Q4-end or Q1 — ChrisGPT · 2026-10-29