GPT-4 stays accurate on unpublished social science studies and open-weight models also perform well
RobbWiller · x · 2026-07-23
Model comparison
The paper reports that GPT-4’s predictions remained strongly correlated with observed effects even when the experiments were not public by the training cutoff or were still unpublished in June 2025.
Robustness
- GPT-4: raw correlation around .92 in the comparison plot
- High correlations held for experiments from multiple social-science domains
- Open-weight models also performed well, with correlations in the .81–.86 range in the figure
Takeaway
The result is not just a memorization artifact: performance stayed high on unpublished or later-published studies as well.
Related event: Nature Study: GPT-4 Can Predict Social Science Experiment Results(18 posts)→
More from Research
- AW-0 hits 80.8% on RoboTwin tasks, beating strongest baseline by 16 points — trevordarrell · 2026-10-27
- AW-0's code stream fixes world model prediction: picks mango where baseline grabs apple — trevordarrell · 2026-10-27
- AW-0 debuts: one generative world action model plans in code, predicts, and controls robots — trevordarrell · 2026-10-27
- Scale can't fix VLM reasoning: EMNLP paper shows reporting bias is the bottleneck — RanjayKrishna · 2026-10-27
- MissionReport benchmark: only 3 of 540 AI-generated government reports pass all criteria — Exp_Mark · 2026-10-27
- Finch 1: a geospatial foundation model trained on imagery from 50+ sensors — Scobleizer · 2026-10-27