Buoyant Open-Sources Kinference, an Inference Control Plane for vLLM on Kubernetes
vllm_project · x · 2026-11-06
The vLLM blog features Kinference, a new open source project from Buoyant that acts as a control plane for running vLLM on Kubernetes. It keeps vLLM as the inference engine and adds platform features on top: token rate limiting, tenant usage and cost tracking, intelligent load balancing, and inference policies. Clients simply point OpenAI- or Anthropic-compatible traffic at its HTTP endpoint. Its inference-aware load balancer routes based on GPU queue sizes, historical performance, and KV-cache contents, sending each request to the vLLM worker that will do the least work — public benchmarks show notable gains over standard Kubernetes load balancing. A getting-started guide is included.
More from Infra
- Google to spend $17B on massive AI data center in Wyoming — Polymarket · 2026-11-06
- Hugging Face burns 800K GPU hours to release free 70K-word MoE training handbook — Thom_Wolf · 2026-11-06
- Coherent revenue jumps 50% to $2.37B, signs ~$20B five-year PhotonLink deal — BenBajarin · 2026-11-06
- Chip designer pitches Apollo on small AI data centers across 20 US states — pstAsiatech · 2026-11-06
- SpaceX data center pledges 40% power discounts and billions in community benefits — Scobleizer · 2026-11-06
- Mystery seller seeks to dump $500 million of data center loans — dinabass · 2026-11-06