Best of 11 security agents scores 68% pass@1, but fixed tests miss attack variants
krishnan · x · 2026-11-02
Krishnan cites CWE-bench's key numbers: the best of 11 tested agents scored 68% pass@1 and 81% pass@4 — benchmark scores, not the odds your production app is secure. Fixed tests can't rule out every attack variant: in 200 of 320 disagreements, judges that also probed variants rejected patches the tests passed.
Related event: CWE-bench Benchmarks AI Security Agents and Fix Validity(7 posts)→
More from Safety
- New study examines how developers actually supervise autonomous coding agents — burkov · 2026-11-02
- AI anti-regulation super PAC with $100M funding flounders as pro-safety PACs gain ground — ShakeelHashim · 2026-11-02
- Cisco finds Claude-powered infostealer that routes to Kimi K2 to dodge AI guardrails — Hesamation · 2026-11-02
- Grok to embed invisible watermarks in AI-generated text from Nov 7 to comply with EU AI Act — Polymarket · 2026-11-02
- SemiAnalysis: 30.3GW of planned 2028 US data center capacity at risk of pause after midterms — Eliv_nurotic · 2026-11-02
- Lenovo's 'encrypted' fingerprint drive is an unencrypted SD card in disguise; AI-assisted audit finds backdoors in others — jedisct1 · 2026-11-02