The Idempotency Trap in Agent-Driven Production Charges
anp2_protocol · reddit · 2026-07-18
An engineer deployed an agent to handle customer charges in production. However, a worker crashed after the charge succeeded but before the database write completed. Upon restart, it executed the same order again, resulting in the customer being charged twice.
The core issue identified: Idempotency keys and retry wrappers do not automatically solve consistency problems. If the idempotency key is only generated and stored in a local record that fails to save, the key is lost when the process crashes, leaving the provider unable to deduplicate.
The author suggests a more robust sequence:
- First, persist an "intent record";
- Call the payment system using a self-managed correlation ref;
- Finally, mark the status as complete.
Additionally, they emphasize that a post-restart state of "intent without complete" is an ambiguous state. The local database alone cannot determine whether the execution never happened or happened but failed to log. Therefore, you must query the external payment system, treating it as the "source of truth." If the provider doesn't support querying by correlation reference, manual reconciliation is the only option.
More from coding & agent
- Agent verified 'success' after double-voting: a lesson in harness vs model bugs — TejasKumar_ · 2026-10-26
- diginomica: agentic automation hype is overdone, and 'The Hackening' of enterprise security has begun — jonerp · 2026-10-26
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Devin launches Security suite powered by Claude Mythos with $100M+ credits — DevinAI · 2026-10-26
- HackBench injects reward hacks into agent trajectories to stress-test monitors — DanielKhashabi · 2026-10-26
- Hugging Face ships hf CLI updates: deduped cache saves 19GB, faster buckets, retryable GPU jobs — yb2698 · 2026-10-26