Kimi K3 uses bounded decay to keep chunkwise KDA in BF16 range
stochasticchasm · x · 2026-07-28
The attached figure shows a method-level comparison between Kimi Linear and Kimi K3.
Main points:
- Kimi K3 replaces an unbounded negative-softplus log-decay with a lower-bounded sigmoid parameterization.
- The bound is fixed at gmin = -5, which keeps the reciprocal rescaling factor within BF16 range.
- Because the decay stays in a finite range, both diagonal and off-diagonal tiles can use dense Tensor Core matrix multiplications.
- That removes the special position-pair diagonal path and makes chunkwise KDA computation more efficient.
In short, the design trades an unconstrained decay for a bounded one to improve numerical stability and compute efficiency.
More from Infra
- AMD says Instinct MI455X will deliver 34x MI355X token throughput — Beth_Kindig · 2026-07-28
- Google Cloud adds near-real-time billing anomaly alerts for Gemini API and Vertex AI — rseroter · 2026-07-28
- Block Attention Residuals cuts attention overhead from O(Ld) to O(Nd) — stochasticchasm · 2026-07-28
- LlamaIndex releases create-llama-worker to deploy LlamaParse on Cloudflare Workers — llama_index · 2026-07-28
- OpenRouter shows provider-level pricing, latency, and routing modes for the same model — gnukeith · 2026-07-28
- Kimi K3 serving uses relative GPU queue discounts to boost load balancing — Abhishekcur · 2026-07-28