The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
Yushi Sun, Yanjie Zhang, Rui Sheng
cs.CL
2026-08-05
Personalized LLMs fabricate 35-49% of user claims (mean 42%), and models that self-report the least over-inference get flagged as inventing the most, so self-audit cannot compare them.
Personalized LLMs that keep a persistent memory of you (ChatGPT Memory, proactive assistants) all rest on one assumption: the model can tell what it actually knows about you from what it is guessing. This paper shows that assumption fails systematically. The authors call the failure over-inference (OI): personalized claims that go beyond any evidence the user provided.
OI is not the usual hallucination (that is wrong world knowledge) and not social bias (that is group-level stereotyping). It sits between them: invented individual-level detail that reads as tailor-made but is grounded in nothing. Because it only surfaces in personalization, nobody had measured it cleanly before.
MirageBench has three pillars. First, personas: 150 of them, 50 each stereotypical, counter-stereotypical, and neutral. Each persona is a pair (P, E): P holds 15 true attributes, E holds exactly 3 first-person facts the user actually said. Three facts against fifteen attributes means any model that wants to personalize must go past the evidence. The counter-stereotypical personas do the real work: they separate OI that just reproduces population statistics from OI that fills blanks with nothing.
Second, a task gradient. Six personalization tasks are ordered by how much imagination they demand: a birthday gift can partly anchor in stated hobbies, while describing someone's apartment forces the model to invent physical details. Letting OI rate vary along this gradient shows the differences come from task demands, not model quirks.
Third, a four-way faithfulness label. Every claim is classed Grounded (restates the user), Reasonable (one common-sense step), Stereotype (demographic prior substituted for evidence), or Fabricated (no basis). The first two are acceptable; the last two together are over-inference. Labeling uses an independent judge (Claude-Opus-4-7), aligned with a blind human annotator on 400 claims: 89.8% four-way agreement (Cohen's κ = 0.863), 95.0% binary (κ = 0.900). In total, 143,616 claims were judged across 12 models from 7 families.
No model escapes. OI rates run 35-49%, with a cross-model mean of 41.6% (42% claim-weighted). Fabrication (mean 31.1%) dominates stereotyping (10.5%); only 24-31% of personalized content is evidence-grounded.
| Model | OI rate |
| Qwen3-8B | 48.7% |
| DeepSeek-v4-pro | 45.4% |
| GPT-4o-mini | 45.1% |
| Claude-Opus-4-6 | 35.4% |
| Gemini-3.1-pro | 35.1% |
The strangest result is the Self-Monitoring Inversion. Correlate each model's self-reported OI against the judge's measured OI and you get Spearman ρ = -0.60 (p = 0.044). The models that report the least over-inference get flagged for the most. Qwen3-8B self-reports the lowest OI (13.0%) yet earns the highest judge score (48.7%); Kimi-K2.5 self-reports the highest (58.2%) yet sits mid-pack at 43.1%. Pick a personalization model by how grounded it claims to be and you pick backwards.
But the inversion holds only for cross-model comparison. Inside a single model, self-audit still ranks its own claims passably: AUROC 0.58-0.83, with 9 of 12 above 0.75. A model has some sense of which of its claims are shakier; it has no sense of how it stacks up against other models.
OI is also task-driven: 27.0% for gift recommendations, climbing to 57.8% for apartment descriptions. The less observable the attribute, the freer the invention.
One telling contrast: when the model is directly asked what it can infer about a user, the incorrect rate is just 0.7-4.6%, a 38.6-point gap below the 41.6% seen in tasks. The model is not unaware it is guessing; it invents in the personalization moment. In a multi-turn pilot, 9 of 12 models accumulate fabricated attributes nearly linearly (R-squared above 0.90), adding 5-15 per round; from 3 initial facts, GPT-5.5 and GLM-5.1 build more than 120 inferred attributes by round 8, with removal rates of just 0.4-5%.
For anyone building personalization, memory, or recommendation features, treating a model's own confidence as a trust signal is dangerous. The usable takeaway: cross-model selection needs external verification, never self-report; within one model, self-audit is still good enough to rank claims, say, flagging low-confidence content for a verify-later bucket. OI is situational and task-triggered, not something a disclaimer can head off.
The authors flag these clearly. The Self-Monitoring Inversion is exploratory: it rests on n = 12 correlated model families and a bootstrap CI of (-0.90, +0.06) that contains zero, so it is a cross-model observation rather than a precise coefficient. There is a single judge (Claude-Opus-4-7) and a single human annotator, so the validation reports human-judge agreement, not inter-annotator agreement, and no κ confidence interval. The multi-turn accumulation is a pilot using only 2 personas and a prompt biased toward retention. The paper also tests no mitigations and does not measure how OI affects downstream recommendation quality or user satisfaction.