Agent tests stayed green while it refunded a customer twice
tom_mathews · reddit · 2026-11-15
The author built a synthetic support agent (order lookup, refunds, emails) to stress-test agent testing itself, planting a classic payment failure: the API commits the refund, then times out. The agent retries without an idempotency key, refunding the customer twice — while replying politely and correctly ("Your refund of $84.00 has been processed").
Key insight: every test passed because they all asserted on the reply; the bug lived in tool calls and the payments ledger.
What finally caught it:
- counting tool calls per business key (at most one refund per order id);
- asserting on end system state, not just transcripts;
- recording the incident run and replaying it deterministically in CI;
- deliberately mutating recorded runs (duplicate/drop/reorder tool calls) — output-only tests missed all 29 mutations.
The author, who maintains an open-source framework around this approach, asks how others assert on side effects, handle retries/idempotency, and keep CI deterministic and cheap.
More from coding & agent
- Anthropic designer Nate Parrott on Claude Design workflows and 5 tips to avoid AI slop — petergyang · 2026-11-15
- Perplexity agent paints a macaw in 20,068 brushstrokes with a JS Canvas engine — jeffgrimes9 · 2026-11-15
- The 6 attack surfaces to test in every customer-facing agent — vivvvekredddit1122 · 2026-11-15
- Developer ditches ChatGPT desktop for T3 Code after third try, citing frictionless multi-model switching — aniketapanjwani · 2026-11-15
- DeepMind's AlphaProof Code formally verifies Signal crypto with 6M lines of Lean in 4 weeks — dl_weekly · 2026-11-15
- Which steps need an agent and which should stay fixed? A booking breakdown — AlarmingPepper9193 · 2026-11-15