NBER paper: LLM workflow flags discrepancies in 3,460 of 4,452 published economics replications
RexDouglass · x · 2026-09-29
An NBER working paper (w35782) by Matthew Schwartz, Isaiah Andrews, and Jesse M. Shapiro introduces an open-source LLM workflow that reproduces, improves, and extends published economics research using their replication packages.
- Reproduce: reruns original calculations, checks discrepancies, and runs automated sensitivity analysis. Across 4,452 replication packages from five economics journals, it flagged discrepancies in 3,460 articles or appendices.
- Improve: using alternative implementations, it cut computation time by more than 10x at similar or greater accuracy in 496 articles.
- Extend: it developed novel extensions consistent with the original articles' goals in 923 articles.
LLMs were used in the analysis and writing; Schwartz worked as an Anthropic contractor, but the paper's views aren't endorsed by Anthropic.
Related event: NBER Paper: LLM Workflow Flags Issues in 3460 of 4452 Economics Studies(3 posts)→
More from Research
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- Kind ICLR Reviewer Raises Score Despite Unresolved Concerns, Citing the Paper's Value — MarkSchmidtUBC · 2026-11-16
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16