45 expert scientists stress-test AI reviewers against Nature-family peer reviews
windx0303 · x · 2026-09-20
A new arXiv paper with 58 authors, including Graham Neubig, had 45 expert scientists evaluate AI reviewers' assessments of Nature-family paper reviews, mapping what AI reviewers do well, where they fail, and what remains. Context: AI reviewers are being deployed in peer review amid polarized views and little evidence. Neubig comments the technology is already close—it's ultimately an institutional problem of how to incorporate it.
More from AGI Musings
- Ben Bajarin: Agentic AI will spawn an 'agentic native' CPU tier in datacenters — BenBajarin · 2026-09-20
- Jaron Lanier: AI consciousness is a question of faith, not science — and that taboo is a problem — iamtrask · 2026-09-20
- Debate: Does longtermism let AI doomers ignore probability entirely? — gabriel_weil · 2026-09-20
- The 'War of the Worlds' panic was likely anti-radio propaganda — and the same logic drives AI doomer media coverage — AlexTensor · 2026-09-20
- Neil Chilson: longtermism lets AI doomers dodge probability arguments — neil_chilson · 2026-09-20
- Sam Altman and Elon Musk banter over a Kardashev 3 civilization as the endgame — beffjezos · 2026-09-20