Security-agent acceptance tests: block attacks, keep legit use working, survive unseen variants
krishnan · x · 2026-11-02
A practical acceptance-testing framework for security AI agents, built around three questions: can the agent block the attack, keep legitimate use working, and survive an independently designed variant test — all within the same repair budget?
Concrete practices:
- Include ordinary customer behavior in acceptance tests
- Add attack variants the agent has never seen, designed by someone else
- Count a disabled feature as a regression, even when the tested attack now fails
Related event: CWE-bench Benchmarks AI Security Agents and Fix Validity(7 posts)→
More from coding & agent
- Math check: 1000 agents succeed exactly as often as 1 — swarms buy latency, not intelligence — CatAstro_Piyush · 2026-11-02
- Running a real-phone agent that posts to TikTok: the hard part is verifying your action actually happened — Orfeo_Lyrakis · 2026-11-02
- WorkOS engineer Zack Proser: I fear untold stories more than AI slop — AI Engineer · 2026-11-02
- Algorand MCP Server launched for blockchain account management and payments — modelcontextprotocol · 2026-11-02
- CorebrumOS Physical AI OS now runs on NVIDIA Jetson with 8 agents — chrismatthieu · 2026-11-02
- Three coding agents debated a design, changed minds, and split the work solo — ashrey-26 · 2026-11-02