LLM peer reviewers put 62% of critiques in paper appendices vs 8.4% for humans
joshgans · x · 2026-11-17
A new study had frontier LLMs (GPT-6 and Claude 5.5) review ICLR 2027 submissions and mapped each critique to the section it came from, compared against human reviewers.
Key findings:
- Human reviewers draw only 8.4% of their criticisms from a paper's appendix; frontier LLM reviewers hit 62%, with critiques spilling past page 30 of the appendix.
- AI reviews skew toward detail audits — more seeds, significance tests, confidence intervals — and rarely address the core idea.
- The pattern holds across API, web chat and coding agents, pointing to the models rather than the interface.
Quote-tweeter Josh Gans frames it as a Polanyi-style tacit knowledge problem: even frontier models have strange ideas about what matters, but it's completely fixable with the right context.
Related event: Study: LLM Peer Reviews Are Biased Towards the Appendix(3 posts)→
More from Models
- Every's vibe check: Gemini 4 Carbon codes better but judgment lags Fable 5.2 — every · 2026-11-18
- 'My favorite model': developers rave about Gemini 4 Carbon as Google re-enters the race — zacharynado · 2026-11-18
- Pre-training trend watch: dropping LR decay for constant LR plus checkpoint averaging — rasbt · 2026-11-18
- Mistral's Le Chonk crowned best model name, free Vibe tier for Europe praised — TheTuringPost · 2026-11-17
- Dev exhausted 128 Claude accounts to dodge usage limits, takes week off — thomasahle · 2026-11-17
- Humans& Launches 744B-Param Collaborator Model Ampersand on Vercel AI Gateway — ericzelikman · 2026-11-17