LLMs significantly outperform trained human reviewers at spotting paper errors
Eliv_nurotic · reddit · 2026-10-29
A medRxiv study (Brian, Pereira Hernández, Yin, Koury & Rust, 2026) inserted errors into 58 clinical trial papers and compared LLM detection rates against human peer reviewers from the original BMJ experiment.
- Human reviewers detected only 22–48% of major errors and 7–45% of minor ones
- Most frontier models hit 95–100% on major errors like missing sample-size calculations, intention-to-treat validity, and unjustified conclusions
- Top models (GPT-6.1 Sol, Opus 5, Fable 5, Opus 5.5) also far exceed humans on minor errors (68–100%)
- Some models (marked †) were tested on a 3-paper subset
Takeaway: LLMs are now substantially better than trained humans at systematically spotting methodological and reporting errors in papers.
More from Models
- Qwen3.5-4B hits ~11.4 tok/s CPU-only on a Ryzen 5 laptop with llama.cpp tuning — solyarisoftware · 2026-10-29
- A video 'made by Opus 5.5' surfaces as Anthropic leak chatter grows — repligate · 2026-10-29
- US models lead China's on use-case post-training and even frontier benchmarks — teortaxesTex · 2026-10-29
- Terra tops Artificial Analysis over Kimi and Fable, sparking benchmark trust debate — teortaxesTex · 2026-10-29
- GPT-Image 2.6 spotted in the wild as OpenAI readies next-gen image model — koltregaskes · 2026-10-29
- Gemini refuses to answer questions about public domain books like Macbeth — stordyr · 2026-10-29