Three acceptance-test rules for piloting security agents, from the CWE-bench thread
krishnan · x · 2026-11-02
Krishnan offers a practical acceptance checklist for piloting security agents:
- Keep ordinary customer behavior in the acceptance tests;
- Add attack variants the agent has never seen, designed by someone else;
- Count a disabled feature as a regression, even when the tested attack now fails.
It echoes CWE-bench's finding that fixed tests can't rule out every attack variant, and repairs must preserve legitimate functionality.
Related event: CWE-bench Benchmarks AI Security Agents and Fix Validity(7 posts)→
More from Safety
- MIT professor warns: 'IRIS Immersion Program' invitation emails are a scam — lucacarlone1 · 2026-11-02
- Dev claims an MCP gives a frontier model 96% false negatives on WildChat — amplifiedamp · 2026-11-02
- AI-text detection study: SEO slop prompts get caught every time regardless of retries — amplifiedamp · 2026-11-02
- Cate Blanchett-Backed Hollywood Group Backs NYT in OpenAI Lawsuit Over Consent — StanfordAILab · 2026-11-02
- New study examines how developers actually supervise autonomous coding agents — burkov · 2026-11-02
- AI anti-regulation super PAC with $100M funding flounders as pro-safety PACs gain ground — ShakeelHashim · 2026-11-02