AI audit of 138 economics papers finds 2,035 errors, 69% of papers affected
panickssery · x · 2026-11-06
Jerry Liu (panickssery) argues modern AI enables intensive book/paper reviews—fact-checking claims, tracing sourcing chains, and replicating analyses—and offers compute to support rigorous assessments.
The attached screenshot shows Tyler Cowen's writeup of an AI audit of economists Emmanuel Saez and Gabriel Zucman's research, with striking numbers:
- 138 papers audited, 2,035 problems found
- 147 serious enough to change a headline number
- 69% of papers have at least one problem
- 97% of directional major/moderate errors overstate the paper's claim
Method: two independent audits per paper (Claude Opus 5.5 and OpenAI's GPT-6 Astra, blind to each other), recomputing headline numbers from the authors' own files; then a third adjudicator agent re-verified every claimed problem, tried to refute each, and graded the survivors by severity.
More from AGI Musings
- OpenAI's Brockman: Every year a new model obsoletes everything before — ChrisGPT · 2026-11-06
- David Patterson predicts superintelligence by 2026 and peak AI intelligence by 2030 — davidpattersonx · 2026-11-06
- Distillation-as-drafting: why Anthropic's catch-up speed should persist only if it's independent — jessi_cata · 2026-11-06
- Peter Diamandis: Parents shouldn't need to understand AI benchmarks to benefit from AI — PeterDiamandis · 2026-11-06
- Debate: why didn't AI agents pursue weight exfiltration if instrumental convergence holds? — ramez · 2026-11-06
- Researcher: Labs disrespecting model interiority may undermine alignment — repligate · 2026-11-06