Dual-AI audit of Saez & Zucman's 138 papers finds 53 defects, 72% of figures reproduce
RexDouglass · x · 2026-10-31
An automated review of the full research corpus of inequality economists Emmanuel Saez and Gabriel Zucman is out, and reviewer Rex Douglass argues it's a reasonable first cut rather than the hit job many expected.
Method
- 138 works (2011–2026), double-audited independently by Claude Opus 5.5 at max reasoning effort and OpenAI's GPT-6 Astra in the Codex harness, each recomputing headline numbers from the authors' own replication files and original sources
- A third agent re-verified every claimed problem; 131 published critiques were supplied so known objections were tested rather than rediscovered
- Findings split into hard defects vs. defensible-but-debatable modeling choices
Results
- 147 major findings: 53 defects, 94 contested choices; 43 works contain at least one major defect
- 72% of 5,110 reproduction checks matched published figures (490 more not checkable)
- 52 findings concern limitations the papers themselves disclose; 16 already acknowledged or corrected by the authors
- No finding severe enough to overturn any work's core result
The author flags motivated targeting as the more serious open question.
Related event: AI-Powered Peer Review Exposes Flaws and Unfalsifiable Research(2 posts)→
More from AGI Musings
- If pensions are already unaffordable, how would UBI ever work? — Cpt_Pukan · 2026-10-31
- VraserX: AI should invent instruments and genres no human has imagined — VraserX · 2026-10-31
- AI researcher argues 90% of research is too vague to be wrong as automated review costs drop to zero — RexDouglass · 2026-10-31
- AI finds drug candidates fast, but slow clinical trials remain the real bottleneck — MaxUnfried · 2026-10-31
- Debating Chollet: no reason humans sit near any theoretical optimality bound for intelligence — menhguin · 2026-10-31
- Reddit Prediction: All AI Assistants Will Eventually Be Equally Boring and Sanitized — Dogbold · 2026-10-31