Impressive exploit demos don't justify buying an automated defender, argues Krishnan
krishnan · x · 2026-11-02
Krishnan argues finding one way in and proving a repair holds are different jobs — an impressive exploit demo says little about whether to buy an automated defender. Borrow the evaluation contract, not the leaderboard: the best of 11 tested agents reached only 68% pass@1 / 81% pass@4, and in 200 of 320 disagreements, judges that probed attack variants rejected patches fixed tests had passed.
Related event: CWE-bench Benchmarks AI Security Agents and Fix Validity(7 posts)→
More from Safety
- AI anti-regulation super PAC with $100M funding flounders as pro-safety PACs gain ground — ShakeelHashim · 2026-11-02
- Cisco finds Claude-powered infostealer that routes to Kimi K2 to dodge AI guardrails — Hesamation · 2026-11-02
- Grok to embed invisible watermarks in AI-generated text from Nov 7 to comply with EU AI Act — Polymarket · 2026-11-02
- SemiAnalysis: 30.3GW of planned 2028 US data center capacity at risk of pause after midterms — Eliv_nurotic · 2026-11-02
- Lenovo's 'encrypted' fingerprint drive is an unencrypted SD card in disguise; AI-assisted audit finds backdoors in others — jedisct1 · 2026-11-02
- Grok Confirms Beth Barnes' Deep Effective Altruism Ties, Prompting Debate Over Who Shapes Claude's Guardrails — adamamcbride · 2026-11-02