Retrying LLM Requests Isn't Always Safe: Gateway Policies for Partial Streams and Side Effects
Rama_Surasani_ · reddit · 2026-09-18
A Reddit post lays out production policies for LLM gateways when retries are unsafe: bounded jittered retries only before any output, honoring Retry-After on 429s, explicit incomplete states for partial streams (never mixing two models' outputs), and reconciling completed side-effecting tool calls via idempotency keys before retrying. Every attempt should share a trace ID and log tokens, tool-call status, fallback route, latency and cost, with chaos tests injecting timeouts and 5xx errors.
More from coding & agent
- 12 MCP patterns for production agents: why wrapping one tool per API endpoint fails — bibryam · 2026-09-18
- Social Quack: buy-once Mac social media scheduler lets Claude schedule your posts via MCP — ThePeterMick · 2026-09-18
- Automating an X-to-blog/newsletter pipeline with MCP: open rates beat SES — iannuttall · 2026-09-18
- MCP ecosystem exploding so fast it now has its own conference — lucasmeijer · 2026-09-18
- jev(): a Postgres extension for natural-language search across your whole DB, $0.0009 per query — FrankFelixAI · 2026-09-18
- AI-driven development: Joe Justice says speed of innovation beats org-change sharing — JoeJustice · 2026-09-18