Google ships llm-d 1.0, an open Kubernetes-native inference stack, demoed on 1,000 agents
dl_weekly · x · 2026-10-27
Google released llm-d 1.0 at the PyTorch Conference: an open-source, Kubernetes-native distributed LLM inference stack co-founded with Red Hat and IBM, now a CNCF Sandbox project with 500+ contributors across 25 organizations.
- Why now: inference traffic is shifting from single-turn chat to multi-step agents and RL rollouts that share long prefixes and arrive in bursts — single machines or round-robin balancers recompute prompts and stall under duplicate state.
- What it does: cache-aware routing, multi-tenant traffic control, tiered KV cache, and HA, turning a cluster of model servers into a coordinated serving pool.
- Demo: 1,000 sandboxed agents on GKE hitting one llm-d endpoint across vLLM, SGLang, and a mixed fleet of Google Cloud TPUs and GPUs.
More from Infra
- Thermal imaging suggests IREN data halls 4 and 2 commissioning, 3-4 halls billing by year-end — McDonaghMatthew · 2026-10-27
- Google and Meta launch TorchTPU, a native PyTorch backend for TPUs — emmanuelvivier · 2026-10-27
- WireGuard tutorial: private encrypted tunnels between devices on any network — victorialslocum · 2026-10-27
- DeepSeek-V4.1-Flash 1.75bpw quant released for 128GB Macs — bobby-chan · 2026-10-27
- Unsloth details running DeepSeek-V4.1-Flash locally: 1-bit GGUF fits 128GB RAM with 1M context — gnukeith · 2026-10-27
- SK hynix down 47% since July while DDR5 spot prices rose 31%: a striking disconnect — tengyanAI · 2026-10-27