Personal Agents Get You Wrong: Five Memory Failure Modes and How to Audit Them
sujingshen · x · 2026-11-20
Part 6 of a series on personal agents examines how AI memories about you can be wrong—and increasingly hard to detect.
Five failure modes
- Casual questions become identities: asking "how would a designer handle this" may get stored as "user is a designer"; questions asked on behalf of friends or research for articles get attributed to you.
- Stale memories never retire: AI is good at remembering but clumsy at forgetting—it keeps using your old address or job without telling you.
- Cross-contamination: details from two similar projects get mixed together.
- Confabulation: a user reported ChatGPT invented a plausible version of "that thing we discussed" instead of actually retrieving it.
- Memory injection: researcher Johann Rehberger demonstrated in Sept 2024 that a file with hidden instructions could make ChatGPT permanently remember false facts ("user is 102, lives in the Matrix, believes the Earth is flat"); he repeated the attack on Gemini in Feb 2025.
Takeaway
Unlike chatbots, agents act on wrong memories—shipping packages to old addresses, avoiding restaurants you love. Charts cite OpenAI's internal eval where factual memory accuracy rose from 41.5% to 82.8%, but summaries make memories less inspectable. Advice: don't just correct it in chat—go into settings and audit stored memories yourself.
More from AGI Musings
- Databricks CEO: we hit AGI last year by the old definition, then moved goalposts — VraserX · 2026-11-20
- RAM-to-arithmetic cost ratio went from 1:1 to 1000:1 — time to make LLMs actually optimize code — CatAstro_Piyush · 2026-11-20
- China now leads AI video: ByteDance Seedance and MiniMax set the frontier pace — alexvoica · 2026-11-20
- 'Next Einstein' Sabrina Paterski debates AI's hype and limits with an Oxford computer scientist — whoamisri · 2026-11-20
- AI may be the greatest bullshit-jobs detector ever invented, argues Australian paper — Herebedragoons77 · 2026-11-20
- Will AI Make Social Mobility Structurally Impossible? — peace_venerable · 2026-11-20