Kimi K3 bounds decay at -5 to keep chunkwise KDA inside BF16 range
suchenzang · x · 2026-07-28
The thread argues that the chunk-wise form of KDA in Kimi Linear remains largely the same, but Kimi K3 introduces a new lower-bounded decay trick to make the math friendlier to finite precision.
What changes
- Kimi Linear uses an unbounded negative-Softplus mapping for log-decay.
- Kimi K3 replaces it with a scaled sigmoid so the log-decay is bounded below.
- The fixed bound gmin = -5 keeps cumulative decay values in a BF16-safe range.
Why it matters
- In Kimi Linear, diagonal tiles still need explicit position-pair computation.
- In Kimi K3, the bounded decay lets both diagonal and off-diagonal tiles use dense Tensor Core matrix multiplications.
- The practical effect is to remove the position-pair diagonal bottleneck in chunkwise KDA computation.
The author notes this is not a totally novel observation about forget gates/weight decay, but it is a useful explanation of how the new parameterization changes the computation path.
Related event: Kimi K3 Optimizes KDA: Prevents Overflow and Upgrades Gate(2 posts)→
More from Research
- Why “open weights” still aren’t the same thing as open source — aronchick · 2026-07-28
- KDA’s low-rank gate doesn’t fix attention sink, author says after scaling it up — stochasticchasm · 2026-07-28
- Kimi K3 paper details SiTU-GLU and quantile balancing for 896-expert MoE — KyeGomezB · 2026-07-28
- Aaron Hertzmann says the brain is not a computer, raising the stakes for AGI — yacineMTB · 2026-07-28
- Stanford researchers use AI to scan 500 million words of state law for red tape — StanfordHAI · 2026-07-28
- Kimi K3’s MoE routing may be driving higher expert-parallel communication costs — stochasticchasm · 2026-07-28