Meta engineers on operating distributed inference at scale: inference needs its own control plane
AI Engineer · youtube · 2026-09-19
Meta engineers Nishant Gupta and Naman Ahuja break down how Meta operates inference at a scale exceeding the world's largest microservices, its fastest-growing workload ever. Key points:
- AI is replaying 2008-style cloud evolution: value migrates upward to orchestration — routing, KV cache management, prefill-decode disaggregation.
- The new challenge is coupling: a routing decision shifts cache hit rate, which shifts batch composition, GPU utilization, and autoscaler behavior. A request is now a distributed transaction; preempting a GPU after 200 streamed tokens can't simply be retried. Reliability must live in the control plane.
- An inference scheduler must see seven axes: GPU generation, memory headroom, KV cache state, warm weights, tenant priority, latency budget, and workflow context.
- Every optimization fits four quadrants: avoid (caching), share (batching), move (routing to cheaper models/regions), or delay (admission control). Optimize cost per successful task, not per token.
- Conclusion: observability is a control loop's input; like VMs needed Kubernetes, inference needs its own control plane balancing latency, cost, and throughput.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Where Residential Proxy IPs Come From: AMA with Massive's CEO — Dylan-Hayes · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04