A 70-experiment benchmark finds GPT-4’s predicted effects at r=.85 versus reality
RobbWiller · x · 2026-07-23
Method and result
The authors evaluated GPT-4 on 70 preregistered experiments from TESS and replications, covering 469 effects and 119,330 participants.
They prompted the model with the study materials and demographic profiles, analyzed the simulated responses, and compared the inferred treatment effects with the real ones. The resulting estimates were highly correlated with the actual effects: r = .85, adjusted r = .92.
Broader pattern
Performance stayed strong across many social-science subfields, not just one narrow task.
Related event: Nature Study: GPT-4 Can Predict Social Science Experiment Results(18 posts)→
More from Research
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27
- DONIC hits 86.79% on bimanual dexterous tracking, crushing prior SOTA's 9.43% — liuziwei7 · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- PSNR below 15 dB in papers likely means the metric was misused, researcher warns — kwangmoo_yi · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27
- Scaling-fit discrepancy: output-head FLOPs are ~65% of compute at the smallest rung, ~3% at 535B — dlwh · 2026-10-27