KubeCon's big theme is inference: a full-stack breakdown from DRA GPU slicing to vLLM and llm-d

AccBalanced · x · 2026-11-16

Inference dominated the conversation at this week's KubeCon, and one thread broke the stack into its production pieces:

GPUs and memory: Dynamic Resource Allocation (DRA) handles GPU slicing (MIG/HAMi), with KEDA scaling pods on queue depth, Karpenter adding GPU nodes, and Kueue rationing GPU quota
vLLM serving: PagedAttention-based KV caching to cut memory waste and boost throughput, with prefill/decode split over NIXL
Inference-aware gateways: Gateway API Inference extensions (e.g. agentgateway) route requests to LLM pods
Distributed scheduling: llm-d sits above vLLM for cache-aware pod selection — though per the diagram, most teams skip this layer
Isolation: agents on this infrastructure need sandboxing

Together it sketches a full inference architecture: agent sandbox → gateway → llm-d → vLLM prefill/decode → GPU nodes.

More from coding & agent