LLM peer reviewers put 62% of critiques in paper appendices vs 8.4% for humans

joshgans · x · 2026-11-17

A new study had frontier LLMs (GPT-6 and Claude 5.5) review ICLR 2027 submissions and mapped each critique to the section it came from, compared against human reviewers.

Key findings:

Quote-tweeter Josh Gans frames it as a Polanyi-style tacit knowledge problem: even frontier models have strange ideas about what matters, but it's completely fixable with the right context.

Related event: Study: LLM Peer Reviews Are Biased Towards the Appendix(3 posts)→

Original post →

More from Models

Models channel →