45 expert scientists stress-test AI reviewers against Nature-family peer reviews
windx0303 · x · 2026-09-20
A new arXiv paper with 58 authors, including Graham Neubig, had 45 expert scientists evaluate AI reviewers' assessments of Nature-family paper reviews, mapping what AI reviewers do well, where they fail, and what remains. Context: AI reviewers are being deployed in peer review amid polarized views and little evidence. Neubig comments the technology is already close—it's ultimately an institutional problem of how to incorporate it.
More from AGI Musings
- Business strategy debate: hide the AI behind humans, not humans behind AI — annetgriffin · 2026-11-12
- If superhuman AI researchers are imminent, will capability problems be easier than safety ones? — ronbodkin · 2026-11-12
- "AI is a terrible teacher": viral pushback against personalized AI tutoring hype — RachelVT42 · 2026-11-12
- OpenAI, Anthropic and DeepMind job posts signal PMs must write evals and code — aakashgupta · 2026-11-12
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- PowerShell creator Jeffrey Snover at Ignite: AI revolution is an extinction event for those who don't adapt — dfinke · 2026-11-12