What failure cases must an LLM gateway pass before automatic failover is safe?
Rama_Surasani_ · reddit · 2026-09-21
An LLM gateway developer (explicitly not promoting a product) asks the community for engineering practices on multi-provider automatic failover, with a systematic checklist.
Failure cases to test first:
- Provider returns 429/503
- Connection times out before any output
- Provider stalls mid-stream
- A side-effecting tool call succeeds but transport fails
- Remaining deadline too short for another attempt
- Fallback lacks the same schema/tool/safety/data-handling capabilities
- Recovering provider causes route flapping
Expected outcomes to guarantee:
- Side effects occur exactly once
- Outputs from different models aren't combined
- Every attempt shares one request/trace ID
- Original request cancelled when possible
- Fallback stays within approved policy
- Latency, token usage, and cost remain visible
The author asks which cases caused real production failures and what teams changed afterward — the comments are where the practical lessons live.
More from coding & agent
- Portracker: open-source self-hosted tool auto-discovers running services and network ports — tom_doerr · 2026-09-21
- Termcp: AI-native terminal platform lets agents drive real interactive CLIs without per-tool MCPs — Psychological-Bid722 · 2026-09-21
- 'Software engineering is dead'? Dev lists databases, K8s, LLM ops that English won't replace — ashishllm · 2026-09-21
- How to turn Claude into your first employee before you can afford one — evielync · 2026-09-21
- Jev guide: the System-1 model that only judges and never writes — xiaohu · 2026-09-21
- Opinion: If an agent can't buy your product, you're building last year's SaaS — diegoposts · 2026-09-21