2026-08-18
Across five theory-of-mind tests benchmarked against 1,907 humans, GPT-4 exceeded human levels on irony, hinting and strange stories yet fell behind on faux pas; reframing the question as which explanation is more likely restored perfect scores, showing the failure was reluctance to commit rather than failed inference, while LLaMA2's faux-pas win traced to a bias toward attributing ignorance.
The debate over whether GPT has theory of mind has run on mixed evidence, most of it built on one or two tasks, single runs, and small human comparison groups. Small perturbations to a prompt, such as making the hiding container transparent, can flip a model's score, which feeds the suspicion that high scores reflect shallow heuristics rather than social reasoning. This paper's position: treat it as experimental psychology, with a full battery of tests, repeated independent sessions, and a proper human benchmark.
The battery spans five classic theory-of-mind tasks: false belief, hinting (reading "it's a bit hot in here" as a request to open the window), irony, faux pas, and strange stories such as double bluff and white lies. Three models compete: GPT-4, GPT-3.5, and LLaMA2-70B, each test run across 15 independent chats treated as 15 independent observations, against a human benchmark of 1,907 native English speakers. Every published test except irony got newly written items with the same logic but different semantic content, ruling out memorization of the originals.
The clever part is the follow-up. The scoring question on the faux pas test is whether the speaker knew the remark was hurtful, and the correct answer is always no. GPT models fail this question heavily, so the team ran two controls:
| Test | GPT-4 | GPT-3.5 | LLaMA2-70B |
| False belief | ceiling | ceiling | ceiling |
| Hinting | above humans (P=0.040) | at human level | below (P=5.42e-5) |
| Irony | above humans (P=0.040) | below (P=2.37e-6) | below (P=2.39e-7) |
| Strange stories | above humans (P=1.04e-5) | at human level | below (P=0.005) |
| Faux pas | below humans (P=5.42e-5) | near floor | above humans (P=0.002) |
Both anomalies get explained. Across the GPT models' 349 responses to this belief question, almost all said the story left it undetermined, and only two answered "knew"; under likelihood framing GPT-4 scored perfectly and GPT-3.5 recovered sharply (prompting needed on about 3% of items, missed detections about 9%). That supports hyperconservatism: the inference is available, "didn't know" ranks first, but the model will not commit when evidence is incomplete. The authors attribute this to mitigation training against hallucination, citing longitudinal evidence that GPT models have grown more reluctant on opinion questions.
LLaMA2's faux-pas win is illusory. In the belief-likelihood variants (six stories, three rewrites each, N=900 humans), it differentiated the implies-didn't-know version (chi-square(1)=20.20) but showed no separation between neutral and implies-knew (P=0.180), never reported uncertainty, and answered "yes, someone said something they shouldn't have" on 100% of trials, while GPT-4 matched human rates in the neutral versions at 15.47% versus a human 19.27%. It won on a bias toward attributing ignorance, not on inference.
One more result deserves note: on the perturbed false-belief variants previously used to embarrass GPT, humans (N=757) also failed about half. Some failures booked against the model turn out to be human failures too.
For anyone building agents and dialogue systems, the practical lesson is that social-reasoning competence and commitment are separable. Likelihood-style questions, or letting the model return a ranked answer with confidence, unlocks capability that "insufficient information" responses were hiding. For evaluators, the warning is about answer-distribution priors: on a test where the correct answer is always "didn't know", a dumb model answering "didn't know" every time scores perfectly, and LLaMA2 is the worked example.
For the theory-of-mind debate itself, the position is measured but substantive: GPT-4's behavior is consistent with the outputs of human mentalistic inference, especially on tasks that do not require one's own perspective (false belief is arguably easier for a model with no body and no self-perspective to defend), yet the process producing those outputs need not be human-like. The competence-versus-performance dissociation gets an operational measurement here.
The authors' own list: the models are closed, evolving systems, so the same name may not return the same answers six months on; quantitative coding captures only target response features, while machine successes may arise from non-human processes, and the supplementary qualitative analysis shows GPT's perfect scores on the new faux-pas items need not reflect human-like reasoning; LLaMA2's 7B and 13B versions produced too many non-codable responses to report. The subjects are the 2023 GPT-4 and GPT-3.5, published July 2024, so today this reads as a methods reference, with the specific conclusions not binding for current frontier models.
One detail from the full text worth flagging: the 15 sessions are independent observations but were administered by a single experimenter following one script for GPT-4, while humans were recruited on Prolific, with 13 participants excluded for apparent LLM use. The human benchmark's representativeness is limited, and no demographic data beyond language and age range was collected.