100-Paper Review: Most Prompt-Injection Defenses Fail Against Adaptive Attacks

Ok-Lab-7347 · reddit · 2026-09-27

An agent developer mapped 100 open-access papers on AI agent security and the evidence is grim for in-model defenses:

Adaptive attacks break nearly everything: "Attacker Moves Second" (Nasr, Carlini et al. 2025) bypassed 12 recent defenses at 90%+ success for most, despite near-zero reported rates; a 500+ participant red-teaming contest succeeded in every scenario; Zhan et al. 2025 broke all 8 defenses tested at over 50%.
Detectors only stop naive attacks: Protect AI, PromptGuard and Model Armor fell at 90%+ under adaptive attacks; on the RedTeamCUA benchmark the best LlamaFirewall/PromptArmor setup caught just 30% of injections.
Safety training doesn't close the gap: Meta SecAlign still attempted 52% of injected actions (32% succeeded) on 50 high-risk tasks while dropping ordinary task completion to 68%. "Mind the GAP" found 219 cases across six models where the model refused in text while attempting the forbidden tool call; prompt wording alone shifted tool-call safety by 21 points.
Guaranteed defenses move the boundary outside the model: Plan-Then-Execute, Dual LLM, and CaMeL (provenance tracking + policy at every tool call). The cost is utility: CaMeL solved 77% of AgentDojo tasks vs 84% undefended.
New attack surfaces: MINJA plants malicious records into agent memory via ordinary queries (90%+ injection, 70%+ hijack in half of setups); a test of 7 major MCP clients found tool descriptions rarely validated, making the LLM the only line of defense.

More from coding & agent