Researcher pushes back on LLM peer review: papers can judge correctness, not significance
sethlazar · x · 2026-09-27
A heated debate over letting LLMs take part in academic peer review. sethlazar argues: switching to a system that is "too new to be verified or validated" is premature; research shows papers are decent at establishing correctness but poor at evaluating significance. He runs an automated newsletter where LLM agents pick papers for his lab and finds their taste "mediocre to crap." His position: once models can do automatic verification, the situation changes — but we should prove we've reached that point before handing over the intellectual ecology, and even then design safeguards proactively. Both sides agree it's a verifiable empirical question best settled by experiments.
Related event: NeurIPS meta-reviewer flags LLM-flooded reviews, sparking academic debate(6 posts)→
More from AGI Musings
- Malthus Saw It in 1798: Wealthy Avoiding Kids, Delayed Marriage Repeats Today — SydSteyerhart · 2026-11-16
- Runway CEO Predicts Fully AI-Generated Cultural Phenomenon by 2027 — c_valenzuelab · 2026-11-16
- Reddit debates the AI scaling curve: everyone says sigmoid, but where's the ceiling? — Successful-War-9050 · 2026-11-16
- '5% worse and 10% cheaper': the button with no vote is already everywhere — eigenrobot · 2026-11-16
- Effective Altruism is top-down and hierarchical, not a grassroots movement — ctjlewis · 2026-11-16
- Lawyers say clients' AI use creates unrealistic expectations and fake case law in settlements — hoofnagle · 2026-11-16