NBER paper: LLM workflow flags discrepancies in 3,460 of 4,452 economics replication packages
sebkrier · x · 2026-09-29
An open-source LLM workflow from NBER working paper 35782 (Schwartz, Andrews & Shapiro) can reproduce, improve, and extend published economics research:
- Reproduction: Across 4,452 published replication packages from five economics journals, the workflow flags discrepancies in 3,460 articles or their appendices, plus automated sensitivity analysis.
- Improvement: In 496 articles it cuts a calculation's runtime by more than 10x at similar or greater accuracy via different implementations.
- Extension: In 923 articles it develops novel extensions aligned with the original paper's goals.
LLMs were used in the analysis and writing; one author worked as an Anthropic contractor, though Anthropic does not endorse the results.
Related event: NBER Paper: LLM Workflow Flags Issues in 3460 of 4452 Economics Studies(3 posts)→
More from Research
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- LatamGPT releases 178.4M-document open Latin American corpus on Hugging Face — OmarUFlorez · 2026-11-16
- Open-source TotalSegmentator spots a 42mm dilated aorta on a routine chest CT — MaziyarPanahi · 2026-11-16