NeurIPS meta-reviewer flags LLM-flooded reviews, sparking academic debate
A Stanford researcher, suragnair, who completed a meta-review for a NeurIPS workshop reported that roughly 1 in 8 papers was largely LLM-generated "slop" and over half of the reviews appeared LLM-written. The post triggered fierce debate, exposing deep divisions over peer review in the LLM era.
Confirmed
- suragnair's observation: 1 of 8 papers was essentially LLM-generated filler, and over half the reviews looked LLM-written; he urged time-pressed reviewers to write even two lines of genuine intuition rather than an LLM-generated review.
- Computational genomics scholar Anshul Kundaje took the opposite stance: LLM-assisted review under human supervision should be allowed, and reviews from the latest models beat the low-quality human reviews he has seen at conferences over many years.
- Kundaje also noted that in open domains lacking verifiable closed loops for frontier models (e.g., biology), LLMs' "pretend understanding" becomes obvious when brainstorming; Andrew Rosen added that LLMs reviewing papers or grants always produce "nice, but…"-style generic critiques.
- Seth Lazar compiled 7 studies on LLM-assisted peer review, arguing the question is empirically testable and already well studied.
Unconfirmed
- The claim that over half the reviews were LLM-generated rests on the meta-reviewer's subjective judgment; no systematic detection method was disclosed.
Why it matters
- The core dispute is whether LLM review is an efficiency tool or a threat to the credibility of peer review. The opposition (sethlazar) argues that peer review reform should not rely on a system that admits to being too new and unvalidated; existing research suggests text-based methods judge correctness reasonably well but poorly assess significance. With both authors and reviewers using LLMs heavily, academia urgently needs consensus on usage boundaries and oversight.
2026-09-27 ~ 2026-09-28 · 6 related posts
- Episode 1: Researchers Find Many Rebuttals Now Written by AI(2026-09-25, 3 posts)
- Episode 2: NeurIPS meta-reviewer flags LLM-flooded reviews, sparking academic debate(2026-09-27, 6 posts)
Primary sources
- Researcher pushes back on LLM peer review: papers can judge correctness, not significance — sethlazar · 2026-09-27
- [source] 7 Key Studies on LLM-Assisted Peer Review, From Bias to Faulty Reasoning — sethlazar · 2026-09-27
- LLM reviewers always find flaws even in great work, Stanford prof says it's pretense of understanding — IanArawjo · 2026-09-27
- [source] NeurIPS meta-reviewer: 1 in 8 papers is LLM slop, over half of reviews AI-generated — suragnair · 2026-09-28
- Over half of NeurIPS workshop reviews look LLM-generated, sparking debate on AI peer review — anshulkundaje · 2026-09-28
- [source] Researcher Kundaje: latest LLM reviews beat the garbage human reviews I've seen for years — anshulkundaje · 2026-09-28