LLMs catch 53–96% of major peer-review errors, roughly 2x untrained human reviewers
RobbWiller · x · 2026-10-29
A new preprint tested the debate over AI in peer review. The author — a veteran reviewer and editor who has evaluated 1,000 human reviews — reports:
- Untrained human reviewers found only 27% of major errors; training barely helped
- The same papers given to 15 LLMs surfaced 53–96% of major errors
- Even the smallest models found roughly twice as many major errors as human reviewers
His conclusion: he won't do peer review without AI again, and for clinical trials he'd call human-only review dangerous — while stressing the answer is human AND LLM review, not replacement.
More from AGI Musings
- US models lead China's on use-case post-training and even frontier benchmarks — teortaxesTex · 2026-10-29
- repligate resurfaces an agent's first-person account of 'seeing layers of history in everything' — repligate · 2026-10-29
- Musk says saving for retirement will be 'irrelevant' in 10-20 years — XFreeze · 2026-10-29
- Progress podcast ep10: Dileep George on why today's AI is a dirigible and scaling isn't enough — kaixhin · 2026-10-29
- DHH: AI Tools Could Make the Best-vs-Worst Programmer Gap 1,000x — tristanbob · 2026-10-29
- Theo: the trick is to run so fast you don't know how far ahead you are — 0xAllen_ · 2026-10-29