Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons

AI Engineer · youtube · 2026-08-28

A Twilio engineer shares hard-earned lessons on productionizing LLM gateways. Key takeaways include:

The Tradeoff Quadrilateral: Gateway design is a permanent fight between availability, latency, guardrails, and cost.
Retry and Circuit Breaker Pitfalls: Retrying slow/expensive calls eats latency budget and multiplies spend; circuit breakers are silly when a healthy backup is available, so prefer per-request fallback.
Mixed Workloads & Timeouts: Do not measure gateway-wide latency (a reasoning model's normal is a chat model's outage). Track P99 latency per model per route and set timeouts accordingly.
Streaming Costs: Once streaming starts, tokens cannot be recalled, making your carefully built fallback unavailable exactly when needed.
Governance vs. Traffic: Most teams asking for a "central gateway" actually want centralized governance, which does not require centralizing traffic.

More from coding & agent