Kimi K3 paper details Attention Residuals and a block design that cuts memory use
suchenzang · x · 2026-07-28
Kimi K3 paper explains Attention Residuals and block-wise memory savings
This thread highlights the paper section on Attention Residuals (AttnRes), which reframes residual connections as learned attention over prior layer outputs.
Main points:
- Standard residuals compress history into one state, similar to an RNN bottleneck.
- Full Attention Residuals let each layer selectively retrieve representations from all previous layers.
- Block Attention Residuals reduce memory and communication overhead by grouping layers into blocks and attending over block-level representations.
- The paper reports that block AttnRes cuts overhead from O(Ld) to O(Nd), and gives a practical example where memory drops from about 15 GB to roughly 1.9 GB per device, or even below 0.3 GB with chunked prefill.
The attached figures also compare cache representations and summarize the I/O cost of different residual schemes.
Related event: Inside Kimi K3: Tri-axis Architecture and Hybrid Attention(34 posts)→
More from Infra
- Compute, not algorithms, is the real moat in frontier AI — GavinSBaker · 2026-07-28
- Renting GPUs and open-weight models cut one AI bill from $1.2M to $100K — kimmonismus · 2026-07-28
- Running Kimi-K3 Locally Costs Up to 1 Million Euros, Jokes German Tech Blogger — FlorianGallwitz · 2026-07-28
- Bittensor updates $TAO emissions with a 61% gate that favors top-valued subnets — markjeffrey · 2026-07-28
- Qualcomm says NPU latency beats GPU and CPU for on-device AI agents — qdrant_engine · 2026-07-28
- MLA-based KV cache costs 12 GB per million tokens, with KDA state at 230 MB BF16 — zephyr_z9 · 2026-07-28