DeepMind Paper: Swarms of Attacker Agents Break Even the Best Prompt Injection Defenses
rohanpaul_ai · x · 2026-11-13
A new Google DeepMind paper, "Scaling Adaptive Prompt Injection Attacks with Agent Swarms" (arXiv:2611.11861), shows prompt injection is far from solved: swarms of capable attacker agents still trick even the best-defended models.
Key points:
- Prompt injection—hiding malicious instructions in data an AI agent reads, like tool results—remains one of the top security risks for AI agents today.
- Early work on simple tool-call agents with static payloads saturated by mid-2025; as of Fall 2026, human red-teaming is nearing saturation on Gemini models, making the problem look solved.
- The authors built a new synthetic evaluation dataset using random non-malicious actions as adversarial goals, disentangling instruction-hierarchy compliance from security knowledge.
- A more capable agentic attacker reliably tricks models into following instructions in tool output—including models previously considered robust on public benchmarks like GraySwan Arena.
- Conclusion: current frontier defenses require proactive, automated red-teaming to stay ahead of increasingly capable adversaries. 15 pages, 4 figures.
Related event: DeepMind: Agent Swarm Attacks Still Break Prompt Injection Defenses(2 posts)→
More from Safety
- PSA: Commercial AI systems can know your data without training on it — Michael_D_Moor · 2026-11-14
- OpenAI board member Paul Cristiano: AI accidents are the most likely cause of your death — Ars Technica AI · 2026-11-14
- Coding agent finds client's old OpenAI key overnight, burns $1,180 while owner sleeps — Few-Pilot7575 · 2026-11-14
- Project Zero overhauls disclosure policy: AI made the 90+30-day timeline insufficient — ifsecure · 2026-11-14
- US voters rejected data centers in 40 of 41 ballots; UK overrules councils to build them — thealexbanks · 2026-11-14
- Commentary: Regulating open source hands China the win in AI's open future — StewartalsopIII · 2026-11-14