Kimi K3 Architecture: Block Attention Residuals and NoPE Integration
nrehiew_ · x · 2026-07-29
Further analysis of Kimi K3 reveals the use of Block Attention Residuals (groups of 12 layers). Traditional residuals suffer from information loss and gradient instability; the new mechanism introduces learnable attention weights and stores summed outputs per block, drastically reducing activation recomputation.
K3 also reuses Kimi Delta Attention (KDA), whose update rule implicitly handles positional representation, allowing it to drop RoPE entirely. NoPE is applied to both KDA and MLA layers (3:1 ratio), with the only change to MLA being an input-dependent gate in the output projection.
More from Research
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16
- Comprehension Audits: New Paper Proposes a Gate for AI-Written Code at Frontier Labs — ronbodkin · 2026-10-16
- Not Boring #215: Cammarata's Clean Networks Claim, Tesla's Flying Roadster, Tau Cut 73% — Not Boring (Packy McCormick) · 2026-10-16