Kimi K3.1 Pro gets Day-0 TPU support: serving a 2.8T-param MoE with llm-d on TPUs and GPUs

vllm_project · x · 2026-11-06

Google partnered with Inferact (vLLM) and RadixArk (SGLang) to deliver Day-0 distributed serving of Moonshot AI's Kimi K3.1 Pro on TPU v7x via llm-d.
The model has 2.8T total parameters (104B active per token across 896 routed experts), a 1M-token context, and a hybrid 93-layer attention stack (69 Kimi Delta Attention + 24 Gated MLA layers); weights exceed single-machine HBM, requiring multi-node tensor and expert parallelism.
llm-d addresses long-horizon agentic serving pain points: 1M-token KV caches exhausting HBM, uncoordinated routing forcing repeated prefix recomputation, and prefill bursts stalling decode — with multi-tenant QoS, zero-downtime upgrades, and a stable v1 API.

More from Infra