ICLR author slams inconsistent LLM review ratings: praise in strengths, clear reject anyway
HildeKuehne · x · 2026-11-10
Researcher Hilde Kuehne shares a pattern she sees across this cycle's ICLR LLM-generated reviews: wildly inconsistent final ratings. Reviews typically praise 'great results' in strengths, then list 3-5 fairly minor weaknesses (e.g., one part of the method lacking novelty) — and then give a clear reject. Each point is reasonable in isolation, she argues, but the weighting and overall judgment are off: LLMs can't assemble the puzzle of a paper's contributions beyond single instances, nor gauge whether a contribution matters to other researchers. Her takeaway: research is still a community sport done by humans, and fully outsourcing review is 'about as attractive as a robot soccer world cup.'
More from AGI Musings
- The Bitter Lesson Misread: Sutton Himself Doesn't Buy the LLM Path — dotey · 2026-11-10
- Comic argues AI scientists will write the next chapter of science, not humans — YogeshMalik · 2026-11-10
- "Alignment" is semantically overloaded — taboo the word to argue clearly — Sauers_ · 2026-11-10
- Creator vows to quit AI-generated content: 'AI really feels like AI, not human' — Abhishekcur · 2026-11-10
- doty clarifies the scaling debate: the point is against hardcoding human priors, not abandoning algorithms — dotey · 2026-11-10
- Type theory researcher tells critics who called verification research useless to admit they were wrong — amplifiedamp · 2026-11-10