CWE-bench Benchmarks AI Security Agents and Fix Validity

On November 2, Krishnan posted a thread introducing the CWE-bench benchmark (preprint dated October 12, arXiv:2610.17020) along with a methodology for evaluating AI security agents.

Confirmed

• CWE-bench consists of 120 "audit-and-fix" tasks built on real open-source repositories with injected vulnerabilities, covering 73 CWE types across 8 languages, and mapped to the OWASP taxonomy
• Among the 11 agents tested, the best result was 68% pass@1 and 81% pass@4; across 320 disputed judgments, there were attack variants that could not be ruled out by fixed tests
• Fix acceptance criteria: a patch must both "block the tested exploits" and "pass all of the repository's existing functional tests"
• Krishnan gave an example: disabling downloads on a file server stops path traversal attacks, but it also removes the point of the service—that doesn't count as a fix

Not Yet Confirmed

• The specific list of agents and their per-task results were not provided in the thread

Why It Matters

• Krishnan repeatedly stressed two points: benchmark scores do not equal the actual probability of security in production; and "finding one attack path" and "proving a fix is robust" are two different jobs—a single impressive exploit demo is not enough to justify a purchase decision
• From this he proposed three pilot acceptance questions: can the agent block attacks, can it keep normal legitimate usage intact, and can it survive variant attacks designed by third parties that the agent has never seen—all within the same fix budget
• Practical recommendations: include ordinary customer behavior in acceptance tests; bring in attack variants designed by others; and even if the tested attacks no longer work, count disabled functionality as a regression

2026-11-02 ~ 2026-11-02 · 7 related posts

• An AI security agent that stops attacks by breaking the app should count as failure — krishnan · 2026-11-02
• [source] CWE-bench: 120 real-repo audit-and-repair tasks test AI defense, best agents hit 68% pass@1 — krishnan · 2026-11-02
• Disabling downloads isn't a fix: CWE-bench requires patches to preserve functionality — krishnan · 2026-11-02
• [source] Best of 11 security agents scores 68% pass@1, but fixed tests miss attack variants — krishnan · 2026-11-02
• Impressive exploit demos don't justify buying an automated defender, argues Krishnan — krishnan · 2026-11-02
• [source] Three acceptance-test rules for piloting security agents, from the CWE-bench thread — krishnan · 2026-11-02
• Security-agent acceptance tests: block attacks, keep legit use working, survive unseen variants — krishnan · 2026-11-02