NBER paper: LLM workflow flags discrepancies in 3,460 of 4,452 economics replication packages
sebkrier · x · 2026-09-29
An open-source LLM workflow from NBER working paper 35782 (Schwartz, Andrews & Shapiro) can reproduce, improve, and extend published economics research:
- Reproduction: Across 4,452 published replication packages from five economics journals, the workflow flags discrepancies in 3,460 articles or their appendices, plus automated sensitivity analysis.
- Improvement: In 496 articles it cuts a calculation's runtime by more than 10x at similar or greater accuracy via different implementations.
- Extension: In 923 articles it develops novel extensions aligned with the original paper's goals.
LLMs were used in the analysis and writing; one author worked as an Anthropic contractor, though Anthropic does not endorse the results.
Related event: NBER Paper: LLM Workflow Flags Issues in 3460 of 4452 Economics Studies(3 posts)→
More from Research
- Anthropic: Claude Autonomously Trains Models to Reliably Mitigate 10 Alignment Failures — inductionheads · 2026-09-29
- What Should an AI Code Reviewer Remember? Persistent Memory vs Chat History — Formal_Cry8001 · 2026-09-29
- Dev Benchmarks Memory Layer vs Vector RAG and Keyword Search: Keyword Tied, Vector RAG Fell — Initial_Orange2985 · 2026-09-29
- Memory poisoning worked 216/216 times on an MCP memory server — until fixes landed — MediaPositive4282 · 2026-09-29
- Palmer Luckey backs $10M XPRIZE to decode animal communication with AI — PeterDiamandis · 2026-09-29
- Self-play on board games distills superhuman CoT, lifting Qwen3-4B math average from 24.1 to 36.6 — PMinervini · 2026-09-29