Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
Mantas Lukauskas, Viktorija Šarkauskaitė
cs.CY, cs.AI, cs.CL, stat.AP
2026-07-07
Asking 37 LLMs to role-play 263 real employees on a 68-item survey, the best model reached a psychometric similarity of 0.71 against 0.69 for a Gaussian-copula baseline with no language model at all; swapping one education field shifted answers by 0.56 SD, and regressors trained on synthetic data scored R2 -0.18 on held-out humans.
Asking an LLM to role-play a survey respondent is now routine in social science: pilot studies, power calculations, and in some published work, outright replacement of human samples. Existing evaluations stop at "do individual answers look plausible" and never ask the measurement-theoretic question: does a synthetic respondent preserve the joint distribution, reliability coefficients, mediation pathways, and demographic effects of real data?
The distinction matters. A synthetic sample with correct item means but wrong item covariances inflates Type I error in any analysis conditioned on a scale score. A sample with correct subscale means but wrong inter-subscale correlations breaks every downstream causal mediation. This paper moves the audit from single items to psychometric structure.
The ground truth is a real Lithuanian organizational-psychology survey: 263 employees, three international instruments (IWPQ, UWES-17, Dunham ATC) plus one locally built scale, 68 items across 12 subscales, with a documented attitudes-to-engagement-to-performance mediation. Lithuania is a deliberate choice: a low-resource language, instruments translated from English and Dutch, so the LLM cannot game the benchmark from English pretraining.
Thirty-seven models (OpenAI, Anthropic, Google, plus 22 open-weight) answer under personas built from real respondents' demographic profiles, 11 fields each, never the person's actual answers. The audit rests on several layers:
About 65,000 synthetic questionnaires back the analysis, under a pre-registered analysis plan frozen before the first LLM call.
| Comparison | Value |
| Human-vs-human ceiling PSS | 0.825 |
| Best LLM (gpt-5.4-mini) | 0.714 |
| Gaussian-copula baseline | 0.688 |
| Correlation component: copula vs best LLM | 0.95 vs 0.52 |
| Discriminator AUC, synthetic vs human | copula 0.39, LLM median 0.999 |
| Synthetic-trained regressor on humans, R2 | -0.18 (human-trained 0.28) |
LLMs do reproduce the direction of human relationships: 36 of 37 models recover the positive mediation, role differences are correctly signed everywhere, and moving from C0 to C3 lifts PSS by 0.18 on average. On the sample-driven components, though, the copula dominates every model: LLM item-correlation matrices are structurally over-coherent, collapsing constructs that humans keep apart. The discriminator result is the bluntest version: copula samples blend into humans (AUC 0.39, below chance) while every LLM is spotted nearly perfectly (median AUC 0.999).
Counterfactual swaps change one demographic field and re-run. Education swaps shift answers by 0.56 SD on average (up to 1.50 on a single subscale), unsupported by the human contrast; role sits at 0.18 and gender at 0.12, the last one fabricated outright given a null human effect. The 0.56 : 0.18 : 0.12 ordering holds across all 37 models.
Two probes close alternative explanations. A verbatim-recall probe bounds item memorization at 4.7% worst case, zero for 22 models, with rank correlation 0.00 against PSS, ruling out regurgitation. And LLMs agree with each other (mean 0.733) more than the best of them agrees with humans (0.71); the wisdom-of-LLMs ensemble collapses to 0.305, below every individual model, because providers err in correlated directions and averaging amplifies the shared error.
For anyone running surveys, policy tests, or instrument pilots, the paper draws a clean line. For direction-of-effect estimates, persona-conditioned LLMs are cheap and adequate. For sample replacement, the answer is no: regressors trained on synthetic respondents score negative R2 on held-out humans, worse than predicting the mean, and an acquiescence shift of +0.84 SD corrupts any analysis that depends on distribution shape.
The methodological lesson generalizes further: anchor LLM evaluations against statistical baselines, add a held-out human ceiling, and attach permutation nulls to similarity metrics. A leaderboard that only ranks LLMs against one another can be fully consistent with all of them losing to a no-language-model baseline. Anyone building agent evaluations or synthetic-data pipelines should steal this design.
The authors list the main ones: a single Lithuanian sample (n=263), so external validity to other countries and constructs is untested; the human reference is self-report, so the LLM-vs-human gap partly reflects human social-desirability bias too; verbatim memorization is ruled out but paraphrased exposure to the English-language instruments is not, so the benchmark tests role-play given possible instrument knowledge, not a never-seen instrument; and at n=100 with one repeat, differences within the top five fall inside the bootstrap CI.
One more caveat from reading the full text: the 37-model lineup froze in June 2026, and the claim that the gap is not closing rests on roughly two years of model generations. The copula's advantage also widens with synthetic sample size (PSS 0.82 at n=200), a comparison that structurally favors the baseline, though it does not change the direction of the conclusion.