Experiment: 15 LLMs Beat Human Peer Reviewers at Catching Errors in Clinical Trial Reports

A medRxiv preprint that drew wide attention on October 29 (Brian, Pereira Hernández, Yin, Koury & Rust, 2026, with USC, ResearchHub Foundation and others involved) used experiments to address the debate over whether journals should ban AI from peer review: 15 large language models comprehensively outperformed trained human reviewers at detecting deliberately planted errors in clinical trial reports.

Confirmed

The experimental design followed an earlier BMJ experiment: 14 major/minor errors were deliberately inserted into 3 test papers and 58 clinical trial reports, and 15 LLMs (including 4 open-weight models, with no tools or internet access) performed the same task as human reviewers.
In the earlier experiment, untrained human reviewers detected on average only about 27% of the major errors in 3 seeded randomized controlled trial reports.
To rule out memorization of older papers, the researchers inserted the same errors into 58 trials published in 2026 (including all eligible open-access Phase III clinical trials from July–October), where AI detection rates ranged from 58% to 99%.
Key supplementary data from the team: 838 of 840 AI reviews beat the human average.

Why It Matters

The results directly challenge current journal policies restricting or banning AI from peer review, providing empirical support for AI-assisted reviewing.
Using newly published 2026 papers rules out data memorization as an explanation, making the conclusions more credible; multiple authors including @anshulkundaje, @RexDouglass, and @rohanpaulai shared and discussed the work, showing strong academic interest.

Note that this work is currently a preprint and has not yet undergone formal peer review.

2026-10-29 ~ 2026-10-29 · 7 related posts

Primary sources

LLMs significantly outperform trained human reviewers at spotting paper errors — Eliv_nurotic · 2026-10-29 05:40 (UTC+08:00)
LLMs catch methodological errors human peer review misses in clinical trials — rohanpaul_ai · 2026-10-29 08:18 (UTC+08:00)
AI reviews catch errors in 2026 clinical trials, beating human reviewers in 838 of 840 cases — RexDouglass · 2026-10-29 06:57 (UTC+08:00)

Preprint: Human peer reviewers caught only 27% of major errors, all 15 AI models beat them (53-96%) — anshulkundaje · 2026-10-29
[source] LLMs significantly outperform trained human reviewers at spotting paper errors — Eliv_nurotic · 2026-10-29
[source] AI reviews catch errors in 2026 clinical trials, beating human reviewers in 838 of 840 cases — RexDouglass · 2026-10-29
Preprint finds AI-assisted peer review beats human-only review; HHMI embraces AI — anshulkundaje · 2026-10-29
LLMs catch 53–96% of major peer-review errors, roughly 2x untrained human reviewers — RobbWiller · 2026-10-29

2 near-duplicate retellings: RexDouglass · rohanpaul_ai