Researcher pushes back on LLM peer review: papers can judge correctness, not significance
sethlazar · x · 2026-09-27
A heated debate over letting LLMs take part in academic peer review. sethlazar argues: switching to a system that is "too new to be verified or validated" is premature; research shows papers are decent at establishing correctness but poor at evaluating significance. He runs an automated newsletter where LLM agents pick papers for his lab and finds their taste "mediocre to crap." His position: once models can do automatic verification, the situation changes — but we should prove we've reached that point before handing over the intellectual ecology, and even then design safeguards proactively. Both sides agree it's a verifiable empirical question best settled by experiments.
Related event: Researchers Clash Over Using LLMs for Peer Review(2 posts)→
More from AGI Musings
- Roko Mijic argues solving the AI Alignment problem might actually be bad — basedjensen · 2026-09-27
- Google turns 28: the new generation are AI-natives, not search-natives — dejanseo · 2026-09-27
- Colossus: the $100B AI factory where renting compute is the least valuable use — mitchdeg · 2026-09-27
- Will fearmongering AI coverage become training data that teaches models to misbehave? — Ghost_Pilot_MD · 2026-09-27
- If IT collapses, the pain won't stop at tech: real estate, dining and retail will feel it too — dhruv2038 · 2026-09-27
- Language is a lossy compression: even the best models train on a thin residue of reality — yunta_tsai · 2026-09-27