vLLM trains a DSpark speculator for 2.8T-param Kimi K3, hitting ~435 tok/s
AccBalanced · x · 2026-09-18
The vLLM team trained a DSpark speculator for Kimi K3, a 2.8T-parameter model, using the DFlash block-level speculative decoding extension from DeepSeek.
Results: single-stream math reasoning jumps from 110 to 435 tok/s/user, and up to 3.5× higher output throughput at matched interactivity under concurrent load. The 5B drafter proposes 8 tokens per step with 6.4 accepted on math.
Training at this scale required multi-node setups: the Speculators library plus a new Mooncake connector streaming hidden states between vLLM inference and training over RDMA—two GB300 NVL72 nodes serving the target, one training the drafter. The recipe is validated on Qwen3.6-35B-A3B, Gemma-4-31B-it, GLM-5.2, and ships in a HF-compatible format vLLM loads directly.
More from Infra
- SurrealDB 4 launches as multi-model ACID database for the agentic era — Scobleizer · 2026-11-04
- Fire at Taiwan Panasonic PCB materials plant may tighten CCL supply further — zephyr_z9 · 2026-11-04
- AMD CEO Lisa Su says revenue to grow over 80% in 2027, data center to top 80% of sales — AccBalanced · 2026-11-04
- Same Claude model served by Bedrock one day and Anthropic the next, dev finds — georgos_pappas · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04
- Gigabyte's AI TOP motherboard packs 2TB RAM and 9 GPUs for home AI labs — vincent_koc · 2026-11-04