ReplaySSM ships in GLM-5.3-Flash and Kimi-K3, speeding decoding up to 2.1x
tri_dao · x · 2026-10-13
- Tri Dao and NVIDIA detail the production rollout of ReplaySSM: it now runs in production in GLM-5.3-Flash and Kimi-K3 and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang.
- The trick is caching SSM inputs instead of the state, which separates output computation from state update during decoding. That yields up to 1.47x faster standard decoding, up to 2.1x faster speculative decoding, and a low-precision SSM state whose rounding error never propagates through the recurrence.
- Tri Dao's take: SSMs always missed inference goodies Transformers get for free (spec decode, quantization) because we weren't caching the right thing. The same IO-aware idea behind FlashAttention, Mamba scan, and paged KV shows many big wins came not from new architectures but from taking memory IO seriously.
More from Infra
- AI Token Economics Debate: Positive Inference Margins vs. the Shale-Oil Bear Case — toptickcrypto · 2026-10-13
- Perplexity built Projects' filesystem as a Git forge for agents, with greps beating semantic search — perplexity_ai · 2026-10-13
- Claude Code wiped a production database via Terraform, destroying 2.5 years of data — aakashgupta · 2026-10-13
- Six Five Pod: Microsoft-NVIDIA PC agents, Marvell's $70B-$90B FY2031 target — pstAsiatech · 2026-10-13
- 800 Blackwells vs 100,000: can labs without GPUs survive the open-weights race? — altryne · 2026-10-13
- Space Data Centers Could Bypass Earth's Bottlenecks, but GPU Networking in Orbit Is the Hard Part — DavidLinthicum · 2026-10-13