LLMs forecast economics RCT effects with 0.054 SD median error, beating naive benchmarks
RobbWiller · x · 2026-11-03
Economist Eva Vivalt's arXiv paper (2610.38883) tests whether LLMs can forecast treatment effects of economics RCTs.
Design
- Source: pre-registered studies from the AEA RCT Registry and Open Science Framework with results first released in 2026 or later
- Only models with knowledge cutoffs before publication were used, ensuring genuine forecasting rather than memorization
- LLMs first extract forecasting questions from pre-registrations, then predict effects
Scale and results
- Forecasts elicited for 889 treatment effects across 39 papers (673 from 33 papers after screening)
- Best model: median absolute error of 0.054 SD, mean absolute error 0.093 SD, MSE 0.0218
- Substantially outperforms benchmarks of forecasting zero effect or the mean effect of contemporaneous papers
Conclusion
LLM forecasts already hold considerable value for informing policy and research decisions.
More from AGI Musings
- What still makes humans unique in the age of AI, per philosopher Pia Lauritzen — AnnaCiaunica · 2026-11-03
- Yoav Goldberg: If you can write a Python program for a task, it's not worth RL-ing over — yoavgo · 2026-11-03
- Alignment debate: treat AI as allies—slow down and stop deceiving them — davidmanheim · 2026-11-03
- Gary Marcus doubles down: Anthropic's AGI projections were 'insane,' $10K bet still on — GaryMarcus · 2026-11-03
- 'AI is just matrix multiplication' is the new 'emotions are just chemicals' — karmicoder · 2026-11-03
- Addy Osmani on keeping critical thinking alive when AI agents do the work — addyosmani · 2026-11-03