Inside OpenAI's inference routing: why the proportional controller had to go
AI Engineer · youtube · 2026-09-19
OpenAI engineers Qianru Lao and Lu Zhang walked through the evolution of their inference load balancer. The old routing weights came from a feedback loop: engines reported signals, a controller smoothed them into scores compared against fleet average, and nudged weights proportionally. It folded many signals into one decision and let constrained engines balance themselves, but nobody could explain why one engine got a higher weight—and tuning one property moved another. Worst was oscillation: traffic shifted off a hot engine cools it, the controller reads cool as spare capacity and sends traffic back, wrecking the KV cache locality routing was meant to protect.
The replacement is a control plane + data plane design: a control plane with a global view of every CPU cluster and GPU engine, and a per-cluster data plane that answers the one synchronous question—which engine serves this request—from a cached snapshot of routing weights. Signals still flow, but into an optimizer minimizing expected end-to-end latency across all traffic, counting network distance and engine-side queueing, under hard constraints that every request is routed and no engine exceeds capacity.
Nearest isn't always right: a region sending 120 req/s at an engine that serves 100 loses to an engine two regions away running 40 of 80 once queue wait is counted. Protections include outlier penalties, retry budgets that tighten as utilization climbs, and load shedding as last resort.
More from Infra
- Leaving DigitalOcean, one site at a time — carnevalem · 2026-09-20
- Google engineers: LLM benchmark harnesses silently drop requests — 200 QPS in, 38 out — AI Engineer · 2026-09-20
- The rig built to run Emacs and doomscroll X is now worth more than its owner's car — tetsuoai · 2026-09-19
- Apple M4 sustains 10 instructions per cycle, beating most rivals; M5 speedup explained — lemire · 2026-09-19
- Apple M6 bumps cores to 12 with two super cores; CPUs keep improving fast — lemire · 2026-09-19
- Apple M-series chips gained ~50% Geekbench 6 performance over three years — lemire · 2026-09-19