GPT-4 Can Accurately Predict Social Science Experiment Outcomes, Nature Study Shows
RobbWiller · x · 2026-08-01
A study published in Nature evaluated the ability of LLMs to predict human behavior. The researchers built an archive of 70 preregistered, nationally representative survey experiments in the USA, comprising 469 experimental effects and 119,330 participants.
By prompting GPT-4 to simulate how American individuals would respond to experimental stimuli, the model's predicted treatment effects showed a very strong correlation (r=0.85) with actual observed effects. Because GPT-4's training-data cutoff predated the publication of many of these studies, the findings suggest the model possesses genuine generalization capabilities rather than merely regurgitating memorized training data.
More from AGI Musings
- Author of 'LLMs feel pain' study tells Gary Marcus: we never claimed that — GaryMarcus · 2026-09-20
- Over 2/3 of a Lab's Paper Reviews Last Year Were Obviously AI-Generated — bclavie · 2026-09-20
- If alignment is not a math problem, where do the random probabilities come from? — inductionheads · 2026-09-20
- The Inference Gap: frontier model access no longer means frontier capability — typewriters · 2026-09-20
- Anthropic Researcher: Alignment Foundations Richer Than the Median of Human Wants — IasonGabriel · 2026-09-20
- Venkatesh Rao: EA Promised to Solve AI Safety — Now We Have Two Problems — round · 2026-09-20