LLM peer reviews draw 50%+ of criticisms from the appendix, humans only 8.4%
TuhinChakr · x · 2026-11-17
An arXiv paper, "AI Reviews Are Biased Towards the Appendix," analyzed 2,000 papers from ACL, EMNLP, COLM and ICLR and found the latest GPT and Claude models draw 50% or more of their review weaknesses from the appendix, versus just 8.4% for human reviewers.
- LLM critiques cluster on fine-grained technical details buried deep in appendices — more seeds, significance tests, confidence intervals — rarely the core idea; human critiques center on presentation and core contributions.
- The pattern holds across API, web chat, and coding-agent access, pointing at the models themselves rather than the interface.
Related event: Study: LLM Peer Reviews Are Biased Towards the Appendix(3 posts)→
More from Models
- Pre-training trend watch: dropping LR decay for constant LR plus checkpoint averaging — rasbt · 2026-11-18
- Mistral's Le Chonk crowned best model name, free Vibe tier for Europe praised — TheTuringPost · 2026-11-17
- Dev exhausted 128 Claude accounts to dodge usage limits, takes week off — thomasahle · 2026-11-17
- Humans& Launches 744B-Param Collaborator Model Ampersand on Vercel AI Gateway — ericzelikman · 2026-11-17
- Claude Opus 5.5 Flagged Iranian School as Military Target in 22 of 32 Tests, Eval Finds — Polymarket · 2026-11-17
- AI models misidentify civilians as military targets in tests, Anthropic worst — djcows · 2026-11-17