AI peer review reads everything better than experts but still judges worse, warns economist
paulnovosad · x · 2026-11-09
Development economist Paul Novosad assesses AI in academic peer review: its weakest link is judgment, for reasons unclear, but it far outperforms human experts at actually reading everything — appendices and pre-analysis plans should be part of evaluation, and AI makes that feasible.
He also flags an enforcement problem: lazy or dishonest academics will submit AI reviews claiming they thought it through themselves. It will be obvious to observers but hard to prove.
More from AGI Musings
- AI safety talent surge debated: is METR's hiring boom really field-wide growth? — austinc3301 · 2026-11-09
- Gary Marcus: soaring costs of incremental AI gains point to a looming bubble burst — GaryMarcus · 2026-11-09
- Open source is easy to start but hard to polish: the real cost is mental endurance — yangyi · 2026-11-09
- AI era favors broad intermediate skills over maxed-out skill trees, argues dev — zack_overflow · 2026-11-09
- Hot take: speeding up individual AI researchers is a weak metric as big labs fight over compute — ZhongRuiqi · 2026-11-09
- Are faster AI researchers a good progress metric? One researcher argues no — ZhongRuiqi · 2026-11-09