ReplaySSM ships in GLM-5.3-Flash and Kimi-K3, speeding decoding up to 2.1x
SonglinYang4 · x · 2026-10-13
Albert Gu amplified the ReplaySSM update, noting that transformer inference has enjoyed dozens of ecosystem-optimized tricks while alternative architectures still hold many untapped optimizations.
Key facts:
- ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang.
- It speeds standard decoding up to 1.47x and speculative decoding up to 2.1x.
- The trick: cache SSM inputs instead of the state, separating output computation from state update during decoding — all three gains flow from this separation.
- It also enables a low-precision SSM state whose rounding error never propagates through the recurrence.
- Blog, paper, and code released in collaboration with NVIDIA and Tri Dao.
More from Infra
- Dual RAID10 NVFP4 setup loads GLM 5.3 Flash cold in ~1 minute — TheZachMueller · 2026-10-13
- Foxconn slides reveal Omniverse digital twins and GR00T humanoids building AI servers — firstadopter · 2026-10-13
- LiquidAI ships MLX versions of d1 models optimized for Apple Silicon local inference — JosephJacks_ · 2026-10-13
- Apple's new Private Cloud Compute release ships vserver images, hinting at virtualized macOS servers — jedisct1 · 2026-10-13
- HF's Zach Mueller: don't do this — most users can just rent or run sandboxes locally on CPUs — TheZachMueller · 2026-10-13
- Celesto CI undercuts GitHub Actions 9x: 1000 jobs for $6.67 vs $60 — aniketmaurya · 2026-10-13