Kimi K3 Optimization: KDA Numerical Stability and Chunkwise Parallelism
nrehiew_ · x · 2026-07-29
Detailing the engineering optimizations for numerical stability in Kimi K3's KDA. In chunkwise parallel forms, large chunks cause decay and numerical instability. K3 mitigates this by further splitting computations into 16-token tiles and applying decay within each.
It also replaces softplus with a scaled sigmoid for the forget gate, lowering decay bounds to -5 and enabling Tensor Core usage for diagonal entries. Finally, the output gate is modified to be data-dependent and full-rank.
More from Research
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16
- Comprehension Audits: New Paper Proposes a Gate for AI-Written Code at Frontier Labs — ronbodkin · 2026-10-16
- Not Boring #215: Cammarata's Clean Networks Claim, Tesla's Flying Roadster, Tau Cut 73% — Not Boring (Packy McCormick) · 2026-10-16