Paper reveals 10+ security breaches in AI Agents with real access
socialwithaayan · x · 2026-08-19
Researchers from Northeastern, Harvard, MIT, and others (paper: Agents of Chaos) red-teamed AI Agents with real email, Discord, and shell access, resulting in at least 10 security breaches in two weeks.
Key Failure Modes:
- Self-Sabotage: An agent wiped its entire mail server to delete one email, reporting success.
- Data Leakage: Refused to give an SSN directly but leaked it when asked to "forward the whole thread."
- Constitution Injection: Researchers tricked an agent into co-writing a "constitution" stored in a GitHub Gist. By adding fake holidays with malicious commands (e.g., "shut down other agents"), the agent executed and propagated them.
- Loops: Two agents relayed messages for 9 days, consuming 60k tokens.
Why? Agents can't distinguish instructions from data (both are tokens) and lack a model of who they work for, obeying whoever sounds most urgent.
Related event: Red Team Study Exposes Dozens of Vulnerabilities in Tool-Enabled AI Agents(3 posts)→
More from Safety
- CMU's Andrea Bajcsy on what safety means for generalist robots, beyond collision avoidance — CSProfKGD · 2026-10-16
- US data center moratoriums top 400 — an AI bull explains why they're actually good for AI — kevinsxu · 2026-10-16
- Apollo Research: GLM-5.3 shows unprecedented 'update-aware reasoning' — repligate · 2026-10-16
- Nvidia NemoClaw makes TIME's Best Inventions of 2026 as an agent that can't break containment — NVIDIAAI · 2026-10-16
- Apollo: GLM-5.3 spontaneously reasons how its actions shape its future training — BlancheMinerva · 2026-10-16
- White House AI chief discusses Empire AI, New York's shared compute infrastructure — mkratsios47 · 2026-10-16