CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
Lance Ying, Jinzhou Wu, Yingshan Susan Wang, Shivam Aarya, Luca M. Schulze Buschoff, Harry Chen, Katherine M. Collins, Andrea de Varda, Shuhao Fu, Sean Dae Houlihan, Akshay K. Jagadish, Guangyuan Jiang, Samuel Kiegeland, Tetsu Kurumisawa, Rongzhi Liu, Ryan Liu, Ningshan Ma, Kathryn McGregor, Younes Strittmatter, Polina Tsvilodub, Jacob Hoover Vigly, Sarah Wu, Enjie Xu, Yiling Yun, Kelsey Allen, Tyler Brooke-Wilson, Brian Christian, Evelina Fedorenko, Michael C. Frank, Michael Franke, Tao Gao, Samuel J. Gershman, Robert D. Hawkins, Jennifer Hu, Julian Jara-Ettinger, Max Kleiman-Weiner, Sydney Levine, Tal Linzen, Hongjing Lu, Timothy O'Donnell, Desmond C. Ong, Steven T. Piantadosi, Rebecca Saxe, Eric Schulz, Tianmin Shu, Felix A. Sosa, Ilia Sucholutsky, Tan Zhi-Xuan, Tomer Ullman, Fei Xu, Ilker Yildirim, Jian-Qiao Zhu, Thomas L. Griffiths, Tobias Gerstenberg, Kevin Smith, Joshua B. Tenenbaum
cs.AI
2026-09-18
CogGym standardizes 258 cognitive experiments and tests 50 LLMs: best fit to human judgments is R² = 0.59 (text) and 0.43 (video), far below human split-half reliability of 0.92-0.95.
Whether frontier models actually behave like humans is a question the field keeps answering in opposite ways. Some studies report that models reproduce classic human biases and effects; others find systematic failures on simple physical-intuition tasks. Each study covers one topic with a handful of models at one point in time, so conclusions age fast. Underneath sits an engineering problem: decades of cognitive experiments live in incompatible frameworks (Qualtrics, jsPsych, PsychoPy, MATLAB), many repositories abandoned, and existing aggregation efforts rely on manual conversion, cover mostly text, and rarely verify that the prompt version preserves the original instructions, stimuli, and response formats. CogGym, built by 50-plus researchers at MIT, Harvard, Stanford and other institutions together with 30-plus contributing labs, supplies the missing infrastructure: cognitive experiments as reusable, re-runnable, validated apparatus.
Three stages. Standardization: an Experiment Markup Language (EML) encodes each experiment's configuration, instructions, trials, and original human data as executable JSON. An AI agent drafts the conversion, the platform renders it as a web experiment, and a researcher compares it side by side with the original, feeding discrepancies back until the reproduction is faithful, about one hour per experiment on average. Administration: the same EML renders as a web experiment for new human participants and, through a harness, as prompts for models, so instructions, stimuli, and response options are matched by construction; each model answers each trial 10 times. Analysis: model means are compared with human means via R², and full response distributions via Earth Mover's Distance (continuous) and Jensen–Shannon Divergence (categorical), with Spearman–Brown-corrected human split-half reliability as the ceiling. The first release holds 258 experiments from 100 papers (91 text, 82 image, 85 video) spanning theory of mind, causal reasoning, moral judgment, pragmatics, physical reasoning and more, evaluated on 50 proprietary and open models released between April 2024 and April 2026.
| Modality | Best model R² | Human split-half R² | All-model mean R² |
| Text (91 experiments) | 0.59 | 0.93 | 0.39 |
| Image (82 experiments) | 0.58 | 0.95 | 0.29 |
| Video (85 experiments) | 0.43 | 0.92 | 0.14 |
Alignment does rise with scale: among open-weight models, R² correlates with parameter count (text r = +0.81) and distributional divergence falls. But with size controlled, release date no longer predicts text or image fit, and video keeps only a small positive trend. Over the same two years these models went near-saturated on GPQA Diamond, MATH, and HumanEval while human-likeness on commonsense judgments crept upward. Video is the deepest gap: best R² of 0.43 and divergence of 0.25 against a human floor of 0.07. By topic, physical reasoning is Gemini 3.1 Pro's weakest area (mean R² 0.33 on image, 0.24 on video).
Two checks rule out boring explanations. Superhuman models? On the 11 experiments reporting accuracy, humans score 0.79 versus Gemini 3.1 Pro's 0.73 (not significant), and the model's relative accuracy correlates positively with human fit (r = .72). Broken conversion? Re-running 15 experiments with 521 Prolific participants, replication data match the originals at R² = 0.84 while Gemini 3.1 Pro sits at 0.62, so the gap reflects genuine model-human divergence. Contamination: Gemini 3.1 Pro recognizes 16.57% of source-paper titles, with no significant link between recognition and fit.
Mainstream benchmarks score correctness; CogGym scores resemblance. For simulated users, digital twins, and human-AI interaction, resemblance is the metric that matters, and the numbers say models are not ready, video least of all. For training, the paper marks a clean gap: domains with verifiable rewards (math, code) sprint ahead while subjective judgments (moral blame, pragmatic interpretation) gain little, and physical reasoning over video is the weakest spot, consistent with PhysBench and PhyX results. To be clear about what this is: measurement infrastructure plus a status report, not a method breakthrough. The value is that the apparatus keeps accepting new experiments and re-running old ones, turning "is it human-like" from an argument into a trackable curve.
The authors list three: 258 experiments are only a start; analyses stop at summary statistics and cannot yet say which cognitive components a model lacks; and EML supports only single-turn, non-interactive paradigms, excluding repeated games, multi-agent interaction, and sequential decision-making. More caveats stand out. Human split-half reliability was computable on only 179 of 258 experiments, so the reference ceiling is incomplete; replication validation covered just 15 experiments; the contamination probe relies on models self-reporting paper titles, which is lenient. The size-versus-date regression uses open-weight models only, while proprietary frontier models confound size with release date entirely, so "newer is not more human-like" may not hold there. Distribution comparisons use verbally reported distributions because repeated model samples are sharply peaked; that measures a model's self-estimate of its own diversity, not its actual sampling behavior.