AI Peer Review Tested: GPT-5.5 Has Highest Recall but Generates Noise
ChenhaoTan · x · 2026-08-14
Discussing the limitations of evaluating AI peer-review tools based on a new benchmark where 100 known errors were planted into psychology papers. The author notes that while zero-shot GPT-5.5 achieved the highest recall, it generated an astonishing 512 comments—about 60% more than other systems.
The author argues that verification quality cannot be reduced to automated metrics like recall alone, emphasizing the need for real user feedback. They highlight that their SAI system achieved higher recall while maintaining a higher hit rate than competitors.
Related event: Multi-model AI Peer Review Catches 93% of Paper Errors(2 posts)→
More from Research
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- DIY LEGO bench tests AI brick-building, GPT-6 Astra beats Opus — TheLegoless · 2026-10-16
- tszzl's theory: Tornado → Apache license → apaches → Southwest → Colorado — tszzl · 2026-10-16
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- A Qwen3 neuron tracks both Colorado and Tornado HTTP code; tszzl offers a theory — tszzl · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16