Meta engineers on operating distributed inference at scale: inference needs its own control plane
AI Engineer · youtube · 2026-09-19
Meta engineers Nishant Gupta and Naman Ahuja break down how Meta operates inference at a scale exceeding the world's largest microservices, its fastest-growing workload ever. Key points:
- AI is replaying 2008-style cloud evolution: value migrates upward to orchestration — routing, KV cache management, prefill-decode disaggregation.
- The new challenge is coupling: a routing decision shifts cache hit rate, which shifts batch composition, GPU utilization, and autoscaler behavior. A request is now a distributed transaction; preempting a GPU after 200 streamed tokens can't simply be retried. Reliability must live in the control plane.
- An inference scheduler must see seven axes: GPU generation, memory headroom, KV cache state, warm weights, tenant priority, latency budget, and workflow context.
- Every optimization fits four quadrants: avoid (caching), share (batching), move (routing to cheaper models/regions), or delay (admission control). Optimize cost per successful task, not per token.
- Conclusion: observability is a control loop's input; like VMs needed Kubernetes, inference needs its own control plane balancing latency, cost, and throughput.
More from Infra
- The rig built to run Emacs and doomscroll X is now worth more than its owner's car — tetsuoai · 2026-09-19
- Apple M4 sustains 10 instructions per cycle, beating most rivals; M5 speedup explained — lemire · 2026-09-19
- Apple M6 bumps cores to 12 with two super cores; CPUs keep improving fast — lemire · 2026-09-19
- Apple M-series chips gained ~50% Geekbench 6 performance over three years — lemire · 2026-09-19
- Inside OpenAI's inference routing: why the proportional controller had to go — AI Engineer · 2026-09-19
- "Normal people can't afford 2x DGX Spark": local AI hardware cost debate — FlolightC · 2026-09-19