Kimi K3 Architecture: Block Attention Residuals and NoPE Integration
nrehiew_ · x · 2026-07-29
Further analysis of Kimi K3 reveals the use of Block Attention Residuals (groups of 12 layers). Traditional residuals suffer from information loss and gradient instability; the new mechanism introduces learnable attention weights and stores summed outputs per block, drastically reducing activation recomputation.
K3 also reuses Kimi Delta Attention (KDA), whose update rule implicitly handles positional representation, allowing it to drop RoPE entirely. NoPE is applied to both KDA and MLA layers (3:1 ratio), with the only change to MLA being an input-dependent gate in the output projection.
Related event: Kimi K3 report reveals training and systems stack(19 posts)→
More from Research
- Kimi K3 Architecture: KV Cache Offloading vs. KDA Recurrent State — zephyr_z9 · 2026-07-29
- Small-model orchestration roughly doubled task completion in a 100-task benchmark — _raydeStar · 2026-07-29
- Paper adds a human-only authorship attestation to a quantum matrix result — burny_tech · 2026-07-29
- Biohub is hiring for an AI wet-lab role to build biology models — proteinrosh · 2026-07-29
- An AI digest scans 92 journals every week and turns them into one RSS feed — Afinetheorem · 2026-07-29
- A weekly PDB-synced leaderboard tracks open cofolding models — rishabh16_ · 2026-07-29