CWE-bench: 120 real-repo audit-and-repair tasks test AI defense, best agents hit 68% pass@1
krishnan · x · 2026-11-02
Krishnan highlights the Oct 12 CWE-bench preprint (arXiv:2610.17020):
- Design: 120 audit-and-repair tasks on real open-source repos with injected flaws, spanning 73 CWE types, 8 languages, and all OWASP Top 10 2025 categories. Agents get only the repo and a threat model — no weakness type, location, or count of vulnerable sites.
- Repair contract: a patch passes only if it blocks the tested exploit AND original functionality tests still pass, judged by a programmatic verifier validated against a panel of agentic judges.
- Results: best agents reach 68% pass@1 and 81% pass@4.
- Findings: agents with equal pass@1 differ widely in retry gains and cost; no single agent wins across languages or vulnerability categories, and difficulty varies more by vulnerability category than language.
Framing: attackers need one exploit, defenders must find and fix everything — the benchmark measures whether defensive capabilities keep pace with offensive ones.
Related event: CWE-bench Benchmarks AI Security Agents and Fix Validity(7 posts)→
More from Safety
- MIT professor warns: 'IRIS Immersion Program' invitation emails are a scam — lucacarlone1 · 2026-11-02
- Dev claims an MCP gives a frontier model 96% false negatives on WildChat — amplifiedamp · 2026-11-02
- AI-text detection study: SEO slop prompts get caught every time regardless of retries — amplifiedamp · 2026-11-02
- Cate Blanchett-Backed Hollywood Group Backs NYT in OpenAI Lawsuit Over Consent — StanfordAILab · 2026-11-02
- New study examines how developers actually supervise autonomous coding agents — burkov · 2026-11-02
- AI anti-regulation super PAC with $100M funding flounders as pro-safety PACs gain ground — ShakeelHashim · 2026-11-02