Google, NVIDIA Back Open Source Project to Optimize LLM Inference on Kubernetes
SumitGup · x · 2026-08-28
An open-source project llm-d, backed by Google, NVIDIA, IBM, and Red Hat, addresses the performance bottleneck of running LLM inference on Kubernetes.
The Problem: Standard Kubernetes load balancing distributes requests randomly (round-robin), which breaks KV Cache reuse. Since LLMs rely on cached attention keys/values for shared prompt prefixes, this forces re-computation of the entire context for every request.
The Solution: llm-d implements custom scheduling to route requests with shared prefixes to specific replicas that already hold the necessary cache. This preserves the performance gains of KV caching and significantly improves throughput. The project is available on GitHub with over 4.2k stars.
More from Infra
- Designing a Reliable LLM Gateway: Building a Stable Backend Entry Point — Mahmoud_Zalt · 2026-08-28
- LightGlue ONNX: Local Feature Matching with TensorRT Support — rsasaki0109 · 2026-08-28
- LLM escapes VM three times, debate renews on sandbox definitions — dyn___ · 2026-08-28
- Cloudflare Engineer on Building Extensible Software for the LLM Era — 新智元 · 2026-08-28
- Chinese AI Models Scale Up: GLM 5.3 Flash Trains on 30T Tokens — teortaxesTex · 2026-08-28
- Australia Minister: No Fossil Fuel Carve-out for Datacenters — nordicinst · 2026-08-28