llm-d 1.0 ships: multi-tenant QoS, zero-downtime updates, prefix-cache-aware routing
_ScottCondron · x · 2026-10-21
Red Hat released version 1.0 of the open-source distributed inference project llm-d at PyTorchCon, with CoreWeave announcing support as a founding member. The release targets production workloads with multi-tenant quality of service, zero-downtime rolling updates, and prefix-cache-aware routing — aimed at scaling distributed inference while controlling AI serving costs.
Related event: Red Hat, Google and partners release llm-d 1.0 open-source inference stack(2 posts)→
More from Infra
- Musk targets 13GW AI compute by 2027 with 440,000-GPU Vera Rubin cluster — scaling01 · 2026-10-21
- VectorDash Rewrites Windows Search in Rust, Adds Local Reverse Image Search and AI Chat — andy2na · 2026-10-21
- SpaceX earnings: compute buildout beats expectations, capex and debt remain the big risk — JOBhakdi · 2026-10-21
- Enterprise AI data pipelines should start with systems of record, argues Craig Mullins — craigmullins · 2026-10-21
- SpaceX's AI business beats expectations as Q3 net income hits $4.4B — TechCrunch AI · 2026-10-21
- Together AI Berlin meetup: n8n and NVIDIA engineers on running open models in production — togethercompute · 2026-10-21