John Horton highlights Nature study: GPT-4 simulates survey experiments and predicts real treatment effects
johnjhorton · x · 2026-10-25
Economist John Horton shares his favorite paper in the LLMs & social science literature: Ashokkumar et al., published in Nature. The team built an archive of 70 preregistered, nationally representative US survey experiments (469 effects, 119,330 participants) and prompted GPT-4 to simulate respondent reactions, inferring treatment effects. Predictions correlated strongly with actual effects, matching pooled human forecasts, and held for studies published after GPT-4's training cutoff — ruling out memorization. Predictions systematically overestimated effect sizes; on 15 megastudies (606 effects) correlations were lower but comparable to expert forecasters. Horton notes the data and code are public and he's running a follow-up with newer models.
Related event: Debate Erupts Over Whether GPT-4 Can Predict Survey Experiment Outcomes(2 posts)→
More from AGI Musings
- Investor argues LLMs make specialists commoditized and generalists will capture the value — StewartalsopIII · 2026-10-26
- AI researcher calls out X users for dunking on people angry about AI-driven layoffs — mervenoyann · 2026-10-26
- Weight-tampering worm scenario for near-frontier models called 'somewhat scary' — teortaxesTex · 2026-10-26
- What if all human-written code goes in .gitignore? Redditors debate dev's future role — michal-rd · 2026-10-26
- Voice-to-voice needs more attention for less information density, OpenAI researcher argues — athyuttamre · 2026-10-26
- Senior devs are in their Kasparov era: chess history says the centaur phase expires — nixiewright · 2026-10-26