Tsinghua's BEMU benchmark: LLM agents fail almost every 'forget this' request
rohanpaul_ai · x · 2026-11-04
BEMU: First Benchmark for Agent Memory Unlearning
A Tsinghua University paper, Can Agents Forget When Asked?, introduces Bemula, the first benchmark dedicated to memory unlearning in LLM agents.
Background: Long-term memory is now a core component of LLM agents. Users increasingly ask agents to delete sensitive or outdated facts (e.g., a home address), but existing memory benchmarks only evaluate writing, retrieval, and organization—never user-requested deletion.
Benchmark design:
- 50 multi-session dialogue instances
- 135 forget requests across 6 types
- 1,742 test questions
- Each dialogue has a control version without requests, so only systems that actually stored the target can earn credit for forgetting
- 5 test suites measure storage, removal of requested info, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks
Results are negative: Across 6 mainstream memory systems on 3 backbone LLMs (plus Full Context LLM on 2), no system passes more than 1.5% of forget requests end to end, and most pass none—systems either retain the info or over-delete unrelated memories.
Related event: Tsinghua's BEMULA Benchmark Reveals AI Agents Almost Never Truly Forget(2 posts)→
More from Safety
- Honeypots Capture New Danfoss Device Attack Wave, Possibly Tied to Recent Campaign — cyb3rops · 2026-11-04
- Hackers Put Hundreds of French Supermarket Freezers into Defrost, Food Destroyed — shashib · 2026-11-04
- UK's £400B AI economy boost estimate mainly came from Google — now slashed by two-thirds — Wired AI · 2026-11-04
- Backdoored fork of Artex, Baidu Agent+ challenge winner, found pulling malicious Docker image — cyb3rops · 2026-11-04
- Why AI systems leak PII: masking treated as text replacement, not data classification — arpit_bhayani · 2026-11-04
- Report: Microsoft, Google, Amazon in talks with Sierra Club, LCV, NRDC on responsible data center framework — asusarla · 2026-11-04