vLLM trains a DSpark speculator for 2.8T-param Kimi K3, hitting ~435 tok/s
AccBalanced · x · 2026-09-18
The vLLM team trained a DSpark speculator for Kimi K3, a 2.8T-parameter model, using the DFlash block-level speculative decoding extension from DeepSeek.
Results: single-stream math reasoning jumps from 110 to 435 tok/s/user, and up to 3.5× higher output throughput at matched interactivity under concurrent load. The 5B drafter proposes 8 tokens per step with 6.4 accepted on math.
Training at this scale required multi-node setups: the Speculators library plus a new Mooncake connector streaming hidden states between vLLM inference and training over RDMA—two GB300 NVL72 nodes serving the target, one training the drafter. The recipe is validated on Qwen3.6-35B-A3B, Gemma-4-31B-it, GLM-5.2, and ships in a HF-compatible format vLLM loads directly.
More from Infra
- Hyperbolic hires quant researchers to build GPU compute as a tradable asset class — YiMaTweets · 2026-09-18
- Crusoe raises $3.9B Series F at $30.9B valuation to fuel AI energy buildout — beffjezos · 2026-09-18
- Periodic Labs details its stack: 4.1x Megatron throughput, frontier-beating science models on 1,300 H200s — hsu_byron · 2026-09-18
- Bonsai quant hits 50 tok/s at 128k context on a 24GB card, letting users run two sessions at once — julianharris · 2026-09-18
- Jeff Dean: a handful of workloads will dominate world compute, 'crying out' for specialized silicon — AccBalanced · 2026-09-18
- Silicon Data chart shows Nvidia GPUs retaining value well above depreciation schedules — AccBalanced · 2026-09-18