ReplaySSM ships in GLM-5.3-Flash and Kimi-K3: caching inputs over states speeds decoding up to 2.1x
SonglinYang4 · x · 2026-10-14
The ReplaySSM team, in collaboration with NVIDIA and Tri Dao, released a paper, code, and blog post. ReplaySSM caches SSM inputs instead of the state, decoupling output computation from state updates during decoding. It now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang.
Three gains
- Up to 1.47x faster standard decoding: SSM state I/O accounts for nearly 40% of decode latency at long context; ReplaySSM roughly halves state memory traffic
- Up to 2.1x faster speculative decoding: expensive state rollback becomes a buffer-pointer move
- Low-precision SSM state: a low-precision checkpoint serves outputs while a high-precision one carries the recurrence, so rounding error never propagates
The core insight: you don't always need a complex algorithm to go faster — just defer your memory writes, bridging elegant SSM theory and GPU memory bottlenecks.
More from Infra
- DigUp: open-source Mac semantic search over photos, PDFs and code hits 3k GitHub stars — xiaohu · 2026-10-14
- Photonic Training of a 1B GPT-2 Without Backprop Lands in PNAS — IgorCarron · 2026-10-14
- Xinjiang's 167 GW green power, 16ms to Chongqing: the grid edge in AI race — 96Stats · 2026-10-14
- Migrating a key API from Node to Bun cut p99 latency 80% and memory 63% — DanielLockyer · 2026-10-14
- Agent traffic is a strange inference workload: long prefix caching beats raw tokens per second — rohanpaul_ai · 2026-10-14
- LightVAE speeds up video VAE decoding up to 2.24x on H100 with minimal quality loss — Shiqiao Gu · 2026-10-14