One retry, one double charge: a transient DB error exposes agent workflow idempotency traps
Common_Dream9420 · reddit · 2026-10-14
A real postmortem: after weeks of stable operation, a transient DB error triggered a workflow retry — Step 2's committed write was picked up by CDC and consumed, Step 3 failed before the durable checkpoint, and the orchestrator retried Step 2, producing a second committed write and event. Result: three customer records in a bad state and one charge processed twice.
Key takeaways:
- A 200 response tells you nothing about final business state or whether an effect ran exactly once
- Any retry over a committed step (background jobs, CRUD workflows) hits this; agents make it worse by generating and shipping such logic fast
- Idempotency keys, outbox patterns, and dedup exist for this — the real question is verifying they hold under the failure sequence
The author is building FetchSandbox around deterministic checks: teams define exact rules (e.g., one committed customer update), the tool injects a controlled failure, observes the retry, and reads actual saved state — returning pass/fail/unmeasured. They're seeking design partners for DB fault injection and CDC-specific testing.
More from coding & agent
- Claude builds its own test tool, fixes DreamPark MR sky-flicker bug in minutes — nptacek · 2026-10-14
- Argon arrives in Google Antigravity but users say it's useless even on the 5x plan — waqasalibrand · 2026-10-14
- CMU paper: a 4B model rewrites an agent's harness and beats the 35B model it learned from — irinarish · 2026-10-14
- Harness ≠ infrastructure: sandboxing and persistence belong to a different layer — blaizedsouza · 2026-10-14
- Anthropic publishes security guide for locking down Claude Managed Agents sandboxes — brada · 2026-10-14
- Dev used Gemini as software architect to build an Apple Watch caffeine tracker — pythononrailz · 2026-10-14