Prompt Injection Detector Leaderboards Mislead: Best Tool Blocks Only 2% of Real Agent Attacks

A systematic evaluation of 15 prompt injection detectors across 3 benchmarks (arXiv 2610.03448) reveals severe benchmark mismatch: the best-ranked detector catches only about 2% of injection attacks in the realistic AgentDojo agent setting at a 1% false positive rate, showing current leaderboards fail to reflect real-world defense capability.

2026-10-12 ~ 2026-10-14 · 2 related posts

• 15 detectors, 3 benchmarks: best prompt-injection detector catches 2% of real agent injections — lulzxdxdxd · 2026-10-12
• Best BIPIA prompt-injection detector catches just 2% of AgentDojo attacks at 1% FP — lulzxdxdxd · 2026-10-14