FULL STORY

AI Audit of Economics Papers: From Splash to Backlash

An AI-powered audit claiming 2,035 errors in Saez-Zucman papers sparked excitement before economists collectively challenged its validity, with an OpenAI researcher arguing AI critique will expose 'research performance'.

2026-10-29 ~ 2026-10-31 · 3 episodes · 20 posts

Episode 1 · AI Audit of 138 Economics Papers Finds 2,035 Issues (2026-10-29, 12 posts)

Tyler Cowen published on Marginal Revolution a large-scale AI audit of the research of economists Emmanuel Saez and Gabriel Zucman: 2,035 issues across 138 papers, 147 severe enough to change core conclusion figures, with 97% of directional errors inflating results and 69% of papers containing at least one issue. Cowen himself called for a thorough reassessment of the work and related policy arguments. The result circulated widely, with many viewing it as a landmark case of AI reshaping academic review, though some scholars questioned the reliability of AI-driven criticism.

Confirmed

  • The audit covered 138 papers by Saez and Zucman and was published by Tyler Cowen on Marginal Revolution. Per posts by IgorCarron, haider1, 1411337 and HarrySurden, it used dual-AI cross-checking: Claude Opus 5.5 and GPT-6 Astra independently flagged 2,035 potential issues, then a third agent attempted to verify or refute the findings.
  • 2,035 issues were found, 147 severe enough to change core conclusions; 97% of directional errors inflated results; 69% of papers had at least one issue.
  • Cowen stated the work and related policy arguments need thorough reassessment, calling this a first step in AI rewriting economic science, while warning of significant shifts in academic standing.

Controversy

  • Per TaniaBabina's repost, scholar @sccath criticized the approach: AI is not good at such evaluations and the methods are opaque.
  • Per nitarshan's repost, commenter owlposting accused Marginal Revolution of wasting a "public good of information" by publishing unchecked AI criticism on a large platform.
  • GarrisonLovely noted collective pushback from economists.

Why it matters

  • lemire remarked that anyone can now dissect a scholar's work, highlighting how AI has lowered the barrier to auditing academic research.
  • Saez and Zucman are leading scholars on wealth inequality and wealth taxes whose conclusions are frequently cited in policy debates; if the audit holds, those foundations may need reexamination.
  • The episode shows both sides of AI academic auditing: a low-cost tool for mass verification, and itself a new flashpoint over methodological reliability.

Unconfirmed

  • The share of the 2,035 issues confirmed by human review is unclear, and details of the dual-AI process and verification criteria are limited in the source posts.

Episode 2 · Economists question AI-driven 'error audit' of Saez-Zucman wealth tax paper (2026-10-29, 6 posts)

On October 29, a website using AI to "error-audit" the Saez-Zucman wealth tax paper drew concentrated criticism from the economics community, with critics ranging from neutral observers to economists academically opposed to the original authors.

Confirmed

  • Economist Paul Novosad posted multiple comments on the AI audit. He argued the practice amounts to "opening fire," and similar reviews could soon hit every paper—while also cautioning that without knowing the base distribution of "errors" in economics papers (the false-positive baseline), the audit's credibility is hard to judge.
  • Novosad shared a screenshot of an issue the AI flagged as "MAJOR": the claim that a 70% figure in the NBER paper came from dividing the difference of two large, uncertain estimates—yet checking against the paper itself, the accusation doesn't hold up.
  • Novosad also recounted his experience asking AI to list a paper's three biggest errors: because AI is poor at judging severity, so-called "major errors" often boil down to vague statements like "the authors rely on assumption X, which may not hold." In his example, the cited interval was as wide as 40–100%. He suggested that if AI can't find real flaws, that actually indicates the paper is solid.
  • User felpix noted that Jason Crawford shared the audit site lamenting that "other research may be equally shaky," but anyone can feed any paper to Claude and get a critical review; this "audit hodgepodge," published without any prompts or raw outputs, was criticized as sloppy.
  • Economist Antoine Levy said that after checking a few papers he knows well, the audit is "mostly BS"; he emphasized that his academic stance is almost entirely opposed to Saez-Zucman—even opponents find the audit quality problematic.

Why it matters

  • The core of this debate isn't whether the wealth tax paper itself is wrong, but the methodological bar for AI-auditing academic research: lacking a false-positive baseline, not disclosing prompts and raw outputs, and poor error-severity grading all let "audits" produce accusations that look authoritative but are unreliable.
  • Critics span different academic positions (including the paper authors' opposition), showing the criticism targets audit quality rather than defending the authors; as AI review tools spread, similar "firing" could hit every paper, and academia needs corresponding norms and verification processes.

Episode 3 · OpenAI Researcher: AI Critique Will Expose Research Theater (2026-10-30, 2 posts)

OpenAI researcher jachiam0 argues that much non-technical research is "research theater," and as AI can now perform deep academic critique, large amounts of pseudo-knowledge will be forced under review.