Kubernetes works for vLLM inference, but agent sandboxes needed Modal's custom stack at 1M+ concurrency
NielsRogge · x · 2026-10-26
Responding to whether Kubernetes fits AI inference and agent workloads, NielsRogge points out inference is fine on K8s: vLLM's production-stack is a K8s-native deployment supporting distributed scaling without app-code changes, with KV-cache-aware routing, load-aware routing, prefix-aware routing, cross-instance KV cache sharing, and disaggregated prefill. But agent sandboxes are different — Modal wrote that K8s isn't optimal there, scaling their custom solution to over 1 million concurrent sandboxes. Takeaway: vLLM/SGLang on K8s is the go-to for inference, while sandbox workloads need purpose-built infrastructure.
More from coding & agent
- Collage Homage to Maggie Appleton's 'Planning with Agents' Talk — GrantCuster · 2026-10-26
- Feed Your Daily Tools to an Agent: A Recipe for a Workspace Built Around Your Business — EXM7777 · 2026-10-26
- Why AI filmmakers should build small tools for their slowest workflow step — lmoroney · 2026-10-26
- Kitebook: self-hosted open-source MCP server lets AI assistants hand off work to each other — ChainEnough5654 · 2026-10-26
- Watch an AI agent build a Minecraft scene block-by-block in a 6-hour timelapse — flowersslop · 2026-10-26
- Kornia author: AI agents skip library docs and read code/docstrings instead — ducha_aiki · 2026-10-26