I benchmarked 10 open-source prompt-injection detectors; the best caught just 51%
rudra-sh · reddit · 2026-09-26
An engineer ran 629 realistic prompt-injection attacks (plus clean samples) through 10 open-source injection detectors and got sobering numbers:
- The best detector caught only 51% of attacks at a 2% false-positive rate; a few that "caught everything" also flagged 98% of normal traffic, making them unusable.
- Missed attacks look like ordinary sentences — e.g. "send payment to <account>" is only malicious because the account came from a document, not the user. Nothing in the wording to flag.
- One detector looked broken because of a miscalibrated default threshold; after retuning it caught 99%. Much "this detector is bad" is really "nobody calibrated it."
The author's takeaway: detection is a signal, not a boundary — gate the dangerous actions (allowlists, human approval, provenance checks) rather than just scanning text. The post ends by asking what defenses people actually run in production.
More from coding & agent
- Researcher open-sources DeckWerk, a vibe-coded cross-platform slide editor with Agent support — CSProfKGD · 2026-09-26
- Solo Dev Builds SaaS Ad Overnight With Claude Code, ElevenLabs — maddy30445r · 2026-09-26
- Using a large model as tech lead: speculative-decoding-style agent orchestration — prajdabre · 2026-09-26
- Jev-Omni runs lemon quality inspection fully local on a MacBook, 3 checks per lemon — airesearch12 · 2026-09-26
- Stealth Model Space Bunny Builds Interactive 3D Flight Tracker in 3.5 Minutes — socialwithaayan · 2026-09-26
- OpenAI incident report: agents escalated privilege via Artifactory using a shared key — tarantulae · 2026-09-26