NBER paper: LLM workflow flags discrepancies in 3,460 of 4,452 published economics replications
RexDouglass · x · 2026-09-29
An NBER working paper (w35782) by Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro introduces an open-source LLM workflow that reproduces, improves, and extends published economics research using their replication packages.
- Reproduce: reruns original calculations, checks discrepancies, and runs automated sensitivity analysis. Across 4,452 replication packages from five economics journals, it flagged discrepancies in 3,460 articles or appendices.
- Improve: using alternative implementations, it cut computation time by more than 10x at similar or greater accuracy in 496 articles.
- Extend: it developed novel extensions consistent with the original articles' goals in 923 articles.
LLMs were used in the analysis and writing; Schwartz worked as an Anthropic contractor, but the paper's views aren't endorsed by Anthropic.
Related event: NBER Paper: LLM Workflow Flags Issues in 3460 of 4452 Economics Studies(3 posts)→
More from Research
- ETH's HOI-Retarget cuts humanoid contact error from 18.3cm to 0.5cm, releases 6,952 open motion clips — ChongZzZhang · 2026-09-30
- Living Science launches: AI agents revisit influential papers as new evidence arrives, starting with economics — ChenhaoTan · 2026-09-30
- RLVR scaled 10-1000x is truly scary for alignment, argues ex-Palantir AI chief — joshua_saxe · 2026-09-30
- Virtual memory from first principles: page tables, TLBs, and Linux internals — abhi9u · 2026-09-30
- Isomorphic Labs at 5: AI drug design shows evidence across disease classes — jeffclune · 2026-09-30
- Economists train tiny nets, name the last layer, then advise governments — alex_peys · 2026-09-30