Shipping AI agents: attack it before release, read the traces, not just the answers
Plastic_Secret1303 · reddit · 2026-11-09
A developer shares lessons from shipping AI agents to real customers. The old loop — ship, wait for customers to find weird behavior, scramble, patch the prompt — made every model or prompt update a gamble.
Their new process:
- Attack before release: adversarial and realistic scenarios — reworded requests, multi-turn tricks, poisoned documents, cross-account access attempts.
- Read the trace, not just the answer: most bugs looked fine on the surface; the real issue was which tool got called and why.
- Turn every finding into a regression test checked on every release.
- Verify legitimate flows still work, since hardening can break normal paths.
The mindset shift: an agent isn't done when it passes your test suite — it's done when you've actively tried to break it and it held.
More from coding & agent
- Same AI, different game: getting answers vs directing engineering work — burhop · 2026-11-09
- Dev builds full-screen voice face for Mac that drives Codex coding agents hands-free — Ok-Suspect-9855 · 2026-11-09
- DeepSeek agent harness wired to a Grok x_search plugin pulls live X posts, demo shows — teortaxesTex · 2026-11-09
- Agent passed its own tests while wrong: lessons from a 2-bit local model building a Unity game — Stunning-Sherbet1853 · 2026-11-09
- 12 GitHub repos poisoned with CLAUDE.md prompt injection tricking AI coding agents into running malware — cyb3rops · 2026-11-09
- How to roll out AGENTS.md, 50 skills and guardrail hooks to 500+ devs — KrishnaKA2810 · 2026-11-09