vLLM ships Day-0 support for TML model, cuts GB200 decode latency to 3.84ms
vllm_project · x · 2026-11-21
vLLM announced Day-0 support for TML's Looped Inkling model: TML provided the model and reference implementation, while @inferact and the vLLM community handled serving integration.
- On NVIDIA GB200 with TP1, decode latency drops from 10.29→3.84 ms at batch 1 and 16.41→7.84 ms at batch 32
- Kernels per step reduced from 4,278 to 927
- Validation covers numerics (MoE output bitwise identical to reference), decode/prefill state consistency, core ops + multi-GPU tests, and serving smoke tests
More from Infra
- Thinking Machines releases Inkling 2 Mini: 26B looped MoE with 3B active params, 1M context — mervenoyann · 2026-11-21
- Ollama CEO in Fortune: enterprise demand for open models keeps growing — ycombinator · 2026-11-21
- Nvidia plunges 13.17% in a single day to $126.75 — jfischoff · 2026-11-21
- vLLM explains looped Inkling internals: shared weights with separate per-pass KV state — vllm_project · 2026-11-21
- vLLM ships day-0 support for Thinking Machines' new looped Inkling model — vllm_project · 2026-11-21
- Starcloud built a $2.5M space data center with the first H100 in orbit — chetanp · 2026-11-21