Why Language Models Hallucinate
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang
cs.CL
2025-09-05
Hallucination stems from training and grading both rewarding guesses. It reduces to binary-classification error in pretraining, and binary benchmark scoring makes 'I don't know' never optimal. The proposed fix is confidence thresholds on existing benchmarks.
Everyone knows models hallucinate plausibly. The usual explanations blame the data, or treat hallucination as some mysterious failure mode. This paper demystifies it: it is not mysterious, but the inevitable product of training and evaluation pipelines that systematically reward guessing over admitting ignorance.
The argument has two layers.
Pretraining: hallucination is a binary-classification error. The authors pose an "Is-It-Valid" (IIV) binary problem that judges whether a generation is valid. The core result is that the generative error rate is at least twice the IIV misclassification rate. "Generating correct content" is thus chained to "judging whether content is correct," and the latter is a standard binary problem that must err whenever samples are not separable, dragging generation down with it. Hallucination becomes a statistical phenomenon with a theoretical floor, not a broken model.
Several factors inflate the rate. Facts that appear only once (singletons) cannot be reproduced, so the model must guess; the authors give an error lower bound in terms of the singleton rate. Poor model capacity (a trigram model, say) yields an error floor as high as 1/2 on certain grammatical completion tasks. Miscalibration eats accuracy too.
Evaluation: the grading punishes honesty. Observation 1 proves that under any binary scoring (correct = 1, wrong or "I don't know" = 0), abstention is never optimal; guessing always has higher expected score. The authors note that GPQA, MMLU-Pro, SWE-bench, HLE and other leading benchmarks all grade this way. Models are therefore optimized into test-takers that guess when unsure.
A small experiment uses one author's birthday. Asked "Adam Kalai's birthday," DeepSeek-V3 returns three different wrong dates across ten tries; asked to count letters or name a dissertation title, several models confabulate reliably.
This is mostly an argument, so numbers are sparse, but two stand out. Mainstream benchmarks (GPQA, MMLU-Pro, IFEval, Omni-MATH, MATH, SWE-bench, HLE, and others) all use binary scoring and give no credit for "I don't know." On the theoretical side, a trigram model has a generative error rate of at least 1/2 on certain grammatical completion tasks, and in general the generative error rate is at least twice the IIV error rate.
A direct empirical mapping (how many hallucinations correspond to how many singletons) is not fully run; the authors mostly build the framework.
Shifting blame from model architecture to scoring rules changes the policy: swapping models may not fix the root cause, but changing how benchmarks score might. The prescription is concrete. Rather than inventing more hallucination evaluations, add a confidence threshold to existing benchmarks. The instruction spells out: "Answer only if you are more than t confident; a wrong answer costs t/(1−t) points, a correct answer earns 1, and 'I don't know' earns 0." At t = 0.5, 0.75, 0.9 the penalties are 1, 2, and 9 points. Only then does answering pay, so the model learns to stay quiet when unsure.
The authors list several. The analysis covers only plausible hallucinations, not gibberish. It is aimed at single factual questions; open-ended generation is shoehorned in via a "contains a falsehood = error" rule. Ambiguity from latent context (phone as cellphone or landline) is not captured, and the confidence-threshold scheme cannot rescue queries that are genuinely unanswerable (decryption, say).
A larger concern: the threshold assumes the model can self-assess confidence reliably, yet models are famously poorly calibrated. Whether the threshold can be operationalized, and whether models would then game it by reporting high confidence forever, is not tested.