The Idempotency Trap in Agent-Driven Production Charges
anp2_protocol · reddit · 2026-07-18
An engineer deployed an agent to handle customer charges in production. However, a worker crashed after the charge succeeded but before the database write completed. Upon restart, it executed the same order again, resulting in the customer being charged twice.
The core issue identified: Idempotency keys and retry wrappers do not automatically solve consistency problems. If the idempotency key is only generated and stored in a local record that fails to save, the key is lost when the process crashes, leaving the provider unable to deduplicate.
The author suggests a more robust sequence:
- First, persist an "intent record";
- Call the payment system using a self-managed correlation ref;
- Finally, mark the status as complete.
Additionally, they emphasize that a post-restart state of "intent without complete" is an ambiguous state. The local database alone cannot determine whether the execution never happened or happened but failed to log. Therefore, you must query the external payment system, treating it as the "source of truth." If the provider doesn't support querying by correlation reference, manual reconciliation is the only option.
More from coding & agent
- Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27
- How I let my cloud Grok Bot code on my Mac mini via Tailscale, no public ports — talkaboutdesign · 2026-10-27
- Context layout is a cost decision: one memory edit can bust your entire agent prompt cache — dscape · 2026-10-27
- Alchemy adds effect/Workflow support across Durable Objects, Workflows and Lambda Durable Functions — samgoodwin89 · 2026-10-27
- Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
- Alchemy makes all ~2,850 AWS bindings work from any cloud, not just S3/Lambda/DynamoDB — samgoodwin89 · 2026-10-27