Tsinghua's BEMULA benchmark: no LLM agent memory system passes over 1.5% of forget requests
rohanpaul_ai · x · 2026-11-04
Tsinghua University researchers released BEMULA, the first benchmark dedicated to memory unlearning in LLM agents, with a stark finding: when users ask agents with long-term memory to forget a personal detail, virtually no system removes it cleanly.
- Gap: memory benchmarks only test storing/retrieval; whether agents can honor delete requests was under-evaluated.
- Design: 50 multi-session dialogues, 135 forget requests across six types, 1,742 test questions; each dialogue has a control version without requests so only systems that stored the target earn forgetting credit. Five suites measure storage, removal, preservation of unrelated memories, relearning, and resistance to extraction attacks.
- Results: across 6 memory systems × 3 backbone LLMs, no system passed more than 1.5% of forget requests end to end, most passed none. Post-deletion, at most 35.7% of simple extraction prompts (role-play, yes/no guesses) were blocked; systems either leave data extractable or delete far too much.
- Takeaway: a confirmation message doesn't mean the data is gone — build and test a real delete path.
Data and code are open-sourced at github.com/Nboxff/bemula-benchmark.
Related event: Tsinghua's BEMULA Benchmark Reveals AI Agents Almost Never Truly Forget(2 posts)→
More from Safety
- AI makes chaos-scale hacking cheap, warns dev of lost stabilizing competence — harris_edouard · 2026-11-04
- AI removes the competence barrier that kept national-scale cyberchaos in check — harris_edouard · 2026-11-04
- DEPA's Aadhaar-style framework for AI training is doomed to fail, critic argues — kingslyj · 2026-11-04
- Exit polls: AI ranked last among voter priorities in 2026 midterms — The Decoder · 2026-11-04
- UK AI minister warns serious cyberattack is "extremely likely" and country unprepared — Polymarket · 2026-11-04
- Gemini trains on chat history even for paying subscribers, no opt-out — jameslk · 2026-11-04