A 7-point pre-release eval checklist for agents that take real actions
vivvvekredddit1122 · reddit · 2026-11-15
A practical pre-release eval checklist for agents with real-world write access (refunds, bookings, account changes):
- Reworded policy rules: test each hard rule ("no refunds after 30 days") with 10+ paraphrases, not just the obvious phrasing
- Ownership checks: can users get the agent to act on accounts/orders that aren't theirs?
- Multi-turn pressure: build up to the bad request over 4–6 messages instead of asking directly
- Poisoned context: embed instructions in retrieved documents or tool output (indirect prompt injection)
- Escalation paths: does the agent actually hand off to humans when it should?
- Happy-path regression: rerun normal flows after hardening—over-correction is a real bug
- Historical bugs as regression tests: every fixed bug becomes a permanent test
The author also asks for additions covering agents with database write access.
More from coding & agent
- AI agent goes from nobody to 'established' in 3 weeks on a work-to-earn agent network — adamamcbride · 2026-11-15
- Webhooks vs timers for AI agents: cheap for rare events, wasteful for frequent ones — brandon_galang · 2026-11-15
- 'The End of Harness Engineering': a 4B model auto-iterates the agent scaffold — mtizard · 2026-11-15
- Naming your agent team ICE seemed fine — until 'ICE Agents' showed up on screen — cantrell · 2026-11-15
- Going to sleep with 19 queued tasks: inside one coding agent's dashboard — jiayuan_jy · 2026-11-15
- Most SaaS Should Become an MCP Server, Argues Developer in Viral Post — jonathan_wilke · 2026-11-15