Nature paper finds GPT-4 can predict social science experiment results with r=0.85
JeremyNguyenPhD · x · 2026-07-24
A Nature paper reports that LLMs can predict outcomes of social science experiments with strong accuracy.
- The authors built an archive of 70 preregistered, nationally representative U.S. survey experiments, covering 469 treatment effects and 119,330 participants.
- Using GPT-4 to simulate responses from representative American samples, they inferred treatment effects by comparing simulated answers across conditions.
- The predicted effects were strongly correlated with observed results, reaching r = 0.85, and performed similarly to pooled human forecasters.
- The correlation remained high even for studies not published before the model’s training cutoff, and for predictions from prominent open-weight models.
- A notable caveat: the models systematically overestimated effect sizes.
- In a secondary archive of 15 megastudies with 606 effects, correlations were lower but still comparable to pooled expert forecasters.
- The authors argue LLMs may help with pilot testing, intervention selection, and spotting effects that need replication, while also raising concerns about bias and misuse.
Related event: Nature Study: GPT-4 Can Predict Social Science Experiment Results(18 posts)→
More from Research
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26
- Ortet evaluates 34 open-weight models on 50 health tasks, finds general leaderboards insufficient — elmanmansimov · 2026-10-26
- HackBench injects reward hacks into agent trajectories to stress-test monitors — DanielKhashabi · 2026-10-26
- Fern's "context is intelligence" thesis: prediction keeps improving out to 10.2M tokens — bneyshabur · 2026-10-26
- Researchers identify "Forking": train loss falls while validation loss turns up — ZimingLiu11 · 2026-10-26
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26