Google, Red Hat release llm-d 1.0: open K8s-native distributed inference stack
rseroter · x · 2026-10-21
At the PyTorch Conference, Google with Red Hat, IBM and others released llm-d 1.0, an open-source Kubernetes-native stack for distributed LLM inference, now a CNCF Sandbox project with 500+ contributors across 25+ organizations.
Why: workloads are shifting from chat to multi-step agents and RL rollouts that fan out, share long prefixes, and arrive in bursts — generic round-robin load balancing causes redundant prefix recomputation, duplicated KV state, and stalled users.
Key features (illustrated in a three-stage architecture):
- Cache-aware, load-aware routing with HA
- Disaggregated prefill/decode pools serving vLLM and SGLang
- Tiered KV cache: accelerator memory ↔ CPU memory ↔ storage
- Multi-tenant flow control with priority routing (Free/Plus/Pro tiers)
Google demoed 1,000 sandboxed agents on GKE hitting a single llm-d endpoint across a mixed fleet of Cloud TPUs and GPUs.
More from Infra
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- DeepSeek open-sources DeepInfer, the vLLM-based engine behind V4.1-Flash — The Decoder · 2026-10-21
- Robotics startup launches Deployments to serve fine-tuned HF checkpoints to robots in real time — arnie_hacker · 2026-10-21
- ByteDance, Alibaba, Tencent order 3M+ Nvidia RTX PRO 5500 cards worth $35B, Reuters reports — SumitGup · 2026-10-21
- TorchTPU demo: unified vLLM & SGLang backend for LLM serving on TPUs — Rob_Mulla · 2026-10-21
- acs infra launches free base model API offering Llama 3.1 8B and 405B for LLM psychology researchers — amplifiedamp · 2026-10-21