Production agents need durable jobs, not in-loop retries, when side effects matter

Delicious-Energy7016 · reddit · 2026-07-29

A production agent that can mutate real systems exposed a reliability trap: retry logic inside the agent loop dies with the container.

In demos, wrapping tool calls with .withretry() looks fine.
In production, long LLM runs can outlive the process; if the container recycles mid-run, in-memory retry state disappears.
That can lead to calls never firing or firing twice.
The fix is to treat side-effecting tool calls as durable jobs: persistent retries, backoff, idempotency keys, and a record of what actually happened.
The author now hands off anything touching money or external state to durable execution instead of retrying in-loop, and asks how others handle it (Temporal, Inngest, queues, or custom infrastructure).

More from coding & agent