2026-08-01
GPT-4 simulates representative samples' responses to infer treatment effects. Its predictions track 70 real experiments as well as or better than pooled human forecasters (r=0.79), but overestimate effect sizes.
Running a social science experiment is expensive: recruit a representative sample, design the stimulus, randomize, and you are easily into thousands of participants and months of work. Researchers have long wanted a cheap prior, an estimate of the treatment effect before the experiment runs, to choose interventions, pilot designs, and flag which findings deserve replication. The hard part is the forecaster: expert predictions are costly and slow, crowd predictions slower still.
This Nature paper treats a large language model as a cheap candidate. The question is whether a model can role-play a representative American sample, answer the stimulus questions of an experiment, and let us back out the treatment effect the real study would have measured. The Stanford team of Ashokkumar, Hewitt, Ghezae, and Willer built an archive of 70 preregistered US survey experiments (469 effects, roughly 119,000 participants) as ground truth.
The pipeline does not ask the model to guess a number. The team feeds GPT-4 the stimulus materials, has it simulate how representative American individuals would respond, converts every response to percentage points (the full scale = 100pp), and weights the per-condition mean by demographics. Some profiles, such as Black profiles, were oversampled to support subgroup analysis, so weighting restores representativeness. Most experiments lack a designated control, so the analysis randomly picks one condition as the reference, computes the predicted treatment effect, and correlates it with the effect estimated from the original data. That random-reference step is repeated 32 times and the median r is recorded. The archive amounts to 305 unique conditions, 129 outcome variables, and 469 treatment effects.
The temporal gap is the point of the design. GPT-4's training cutoff predates many of the studies. If the model still predicts studies it never saw in training, it is generalizing about people, not reciting memorized results.
The headline: GPT-4's inferred treatment effects correlate strongly with the real ones, and it beats the human benchmark. The lay forecasters the authors recruited reached r = 0.79 (0.84 disattenuated) against the true effects; GPT-4 surpassed that, with raw r between 0.62 and 0.85 across subgroups. The result holds for several prominent open-weight models, and it survives on studies published after the training cutoff (out-of-sample).
Two boundaries are clear. First, the model systematically overestimates effect sizes. Second, it predicts how the average person reacts but not who reacts differently: the interaction of the treatment with gender, ethnicity, and party gives raw r of about -0.01, 0.16, and -0.03, essentially noise. On a secondary archive of 15 megastudies (606 effects), correlations drop but stay comparable to pooled expert forecasters.
If the method holds up, it gives social science a low-cost prior: estimate effects before running, screen interventions, pilot designs, and run power analyses; flag for replication any case where the model expects a big effect that the real experiment fails to produce. The authors also surveyed 460 social scientists (about 41% women, 80% white) on likely uses and risks, mapping applications such as pilot testing, intervention selection, and replication triage against risks of bias and misuse.
For AI practitioners the signal runs the other way: a model trained on enough human text has internalized enough implicit knowledge of how people react to match or beat pooled human forecasters on a controlled task.
The authors' own caveats: the model overestimates effect sizes, so taking its outputs as ground truth biases decisions, and there is a misuse risk of mass-producing predictions that support a preferred stance.
The reviewers' critiques carry more weight (Nature publishes the review file as a supplement). First, the metrics are unconventional: the paper leans on raw r and disattenuated r rather than the MAE, MSE, and RMSE standard in machine learning, and it reports no confidence intervals or significance tests, so "beats humans" is largely descriptive. Second, although the authors name raw r as their primary metric, they argue from the larger disattenuated r, which inflates the numbers. Third, and most striking, out-of-sample accuracy exceeds in-sample accuracy, which a reviewer called perturbing; in conventional ML that pattern usually signals data leakage, and the authors do not squarely address it. Code reproducibility was also flagged as broken out of the box.
On balance this is a capability signal worth taking seriously, but GPT-4 predicting social science experiments currently means it estimates the direction and rough magnitude of average effects reasonably well under the lenient yardstick of correlation. It is some distance from reliable quantitative prediction, and it is nearly useless for subgroup differences.