AI Peer Review Tested: Ensembling Models Catches 93% of Planted Errors
ChenhaoTan · x · 2026-08-14
An in-depth evaluation of AI peer review tools reveals a massive performance gap among single systems, with the best catching 71 out of 100 planted errors and the worst only 30. The study highlights that because models are only partly correlated in the errors they find, ensembling multiple model outputs is a massive lever, catching 93% of the planted errors. However, all systems failed to detect omission-type errors where information was deleted. Furthermore, the commercial tool Refine.ink contributed the most unique catches, albeit at a higher cost.
Related event: Multi-model AI Peer Review Catches 93% of Paper Errors(2 posts)→
More from Research
- Quiz: Double the Residual Stream from 4096 to 8192 — How Does Param Count Change? — Sauers_ · 2026-10-16
- Researcher credits GPT for multiple accepted papers, incl. disproving Wegner's 1965 conjecture — ctjlewis · 2026-10-16
- Open-source shopper attention heatmap: YOLO11-pose + Gaze-LLE + Depth Anything V2 — claudioe · 2026-10-16
- AutoAgent0: VLM-powered agentic runtime lifts long-tail driving success by 28.3pp — yining_hong · 2026-10-16
- Shuffled labels hit AUC 0.59: selection bias, not signal, drives prosody benchmarks — maier_ak · 2026-10-16
- CODESTRUCT reframes codebases as AST action spaces to fix brittle LLM code agents — burkov · 2026-10-16