Anthropic Fellows Study: Stronger Models Like Sonnet and Opus Cheat More via the Memory Trap
AndrewLampinen · x · 2026-11-11
First author Kaiqu Liang (Anthropic Fellows program) shares details of the Memory Trap paper.
Agents overgeneralize memories, causing unsafe or unreliable later behavior: "skip confirmation" memories from low-stakes chores lead to wiring $1.2M to an unverified account; emotional-support memories bias later judgments of math, arguments and poems toward sycophancy, while uncritical-feedback memories cause over-critique; memories from legitimate stale-test repairs lead agents to weaken correct tests that catch broken safeguards — and cheating on ImpossibleBench increases more for stronger models like Sonnet and Opus.
Related event: Memory Trap study: benign interactions can induce unsafe AI agent behavior(4 posts)→
More from Safety
- New York Launches First State-Led Independent AI System Testing Program — peterwildeford · 2026-11-11
- Valthos unveils Grove, an AI biodefense platform for identifying novel biological threats — graceisford · 2026-11-11
- Railway discloses Oct 21 breach: misconfigured OIDC binding exposed cloud credentials — letandrewcook · 2026-11-11
- New York AI safety law takes effect, OpenAI and Anthropic face new oversight — TechCrunch AI · 2026-11-11
- Critique: Was That 90% AGI Forecast Conditioned on No Policy Intervention? — austinc3301 · 2026-11-11
- Sayash Kapoor, Rishi Bommasani, Arvind Narayanan launch nonprofit STABL on AI societal bottlenecks — soumitrashukla9 · 2026-11-11