Red Hat's DSpark Speculator Boosts Kimi-K3 Throughput by 3.5x

teortaxesTex · x · 2026-08-14

Red Hat AI released the DSpark speculator model for Moonshot's Kimi-K3, significantly enhancing inference performance.

Performance: Single-stream interactivity jumps from 110 to 435 tok/s/user on math reasoning. Delivers 3.5x output throughput at matched interactivity under load.
Long Context: Thanks to sliding window attention (2048-token window across all 5 draft layers), the model maintains steady acceptance out to 20K context across 10 LongBench domains.
Training: Since Kimi-K3 does not fit on a single GB300 node, training used a multi-node setup, streaming hidden states between disjoint node groups via a Mooncake store.

More from Infra