Dual-AI audit of 138 Saez-Zucman papers finds 2,035 problems, 69% of papers flawed
IgorCarron · x · 2026-10-29
A large-scale AI replication audit of economists Emmanuel Saez and Gabriel Zucman: 138 papers audited, 2,035 problems found, 147 serious enough to change headline numbers, and 69% of papers had at least one. Method: each paper was independently audited by Claude Opus 5.5 and GPT-6 Astra with no sight of each other's work, recomputing headline figures from the authors' own files and original sources, then a third adjudicator agent re-verified and graded every claimed issue. Of major/moderate errors with a clear direction, 97% overstated the papers' claims (e.g. 'inequality is higher' 188 vs 6). Steve Hsu argues frontier models beat human peer review; Igor Carron agrees.
More from AGI Musings
- Ofir Press: AI's jagged capability frontier will keep advancing in non-intuitive ways — OfirPress · 2026-10-29
- Ofir Press: AI's jagged frontier advances unintuitively — math robots beat self-driving cars — OfirPress · 2026-10-29
- Ofir Press thread: math is more concrete and RL-able than SWE — OfirPress · 2026-10-29
- Ofir Press: Millennium Problems is AI's best benchmark; 70% of open math could fall in a year — OfirPress · 2026-10-29
- Musk says AI and robotics output 'will dwarf everything else,' names Google and Nvidia as top picks — elonmusk · 2026-10-29
- levelsio urges X to adopt zero-tolerance lifetime bans for AI replies — damnGruz · 2026-10-29