7 Key Studies on LLM-Assisted Peer Review, From Bias to Faulty Reasoning
sethlazar · x · 2026-09-27
Amid a dispute over whether AI can replace humans in reading papers, researcher Seth Lazar compiled seven studies on LLM-assisted peer review, arguing it is 'a verifiable empirical question' with substantial existing research:
- 'Stop Automating Peer Review Without Rigorous Evaluation'
- AAAI-26 AI Review Pilot: AI-assisted peer review at scale
- ICML 2026 randomized experiment and survey on LLM use in peer review
- LLM-as-a-Reviewer benchmark: reviewer ability, divergence, and prompt-injection resistance
- A large-scale randomized study of LLM feedback in peer review
- Automatic reviewers fail to detect faulty reasoning in papers
- Hidden bias in LLM-assisted peer reviews
The core point: reliability, bias, and security of AI review are being rigorously quantified — claims that 'AI is great at filtering slop' can't shortcut the evidence.
Related event: NeurIPS meta-reviewer flags LLM-flooded reviews, sparking academic debate(6 posts)→
More from Safety
- Pangram's 750-word false-positive bounty went unclaimed for 5 days — nobody wants to gamble $1,000 — ctjlewis · 2026-11-16
- Researcher reproduces Pangram's FP rate but shows a long epic poem can evade the detector — ctjlewis · 2026-11-16
- Bounty hits $3,000 as researcher shows AI text detectors are trivially evadable — ctjlewis · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Fed Governor Michael Barr: Banks Are Already Catching AI Agents Taking Unauthorized Actions in Customer Accounts — Polymarket · 2026-11-16
- US AI export control carves out commercial recommender systems, sparing ByteDance's search algo — zephyr_z9 · 2026-11-16