vLLM day-0 support for looped Inkling: 2.68x faster decode on GB200 with 78% fewer kernels
vllm_project · x · 2026-11-21
vLLM shipped day-0 support for Thinking Machines' looped Inkling model (12 shared core layers, 3 passes, gated cross-pass messages) along with systems optimizations:
- On NVIDIA GB200 TP1, decode latency drops from 10.29ms to 3.84ms at batch 1 (2.68x) and 16.41ms to 7.84ms at batch 32 (2.09x).
- Kernels per step cut from 4,278 to 927 (78.3% fewer).
- Key engineering: shared weights but per-pass KV cache and convolution state kept distinct (indexed by pass), DSpark hidden states captured per layer visit, and fused message handling plus conv/mix kernels reduce work between recurrent visits.
More from Infra
- Thinking Machines releases Inkling 2 Mini: 26B looped MoE with 3B active params, 1M context — mervenoyann · 2026-11-21
- Ollama CEO in Fortune: enterprise demand for open models keeps growing — ycombinator · 2026-11-21
- Nvidia plunges 13.17% in a single day to $126.75 — jfischoff · 2026-11-21
- vLLM ships Day-0 support for TML model, cuts GB200 decode latency to 3.84ms — vllm_project · 2026-11-21
- vLLM explains looped Inkling internals: shared weights with separate per-pass KV state — vllm_project · 2026-11-21
- vLLM ships day-0 support for Thinking Machines' new looped Inkling model — vllm_project · 2026-11-21