Block Attention Residuals cuts attention overhead from O(Ld) to O(Nd)
stochasticchasm · x · 2026-07-28
Block Attention Residuals cuts attention overhead by partitioning layers into blocks
The attached paper excerpt explains a Block Attention Residuals design that partitions the model’s layers into N blocks of S = L/N layers. Inside each block, layer outputs are reduced into a single representation by summation, and attention is then applied only over the block-level representations rather than all layer states.
Key points from the excerpt:
- The method reduces memory and communication overhead from O(Ld) to O(Nd).
- It also bounds inference-time state, helping inter-block results merge better with sequential intra-block partial sums.
- The authors say this can significantly reduce inference cost.
- Empirically, N ≈ 8 recovers most of the benefit across scales.
- For Kimi K3, they partition layers into 8 blocks of 12 layers, with a partial final block and 9 total blocks if counting the embedding layer.
More from Infra
- vLLM Collaborates with DigitalOcean to Host Kimi K3 Model — vllm_project · 2026-07-28
- AMD says Instinct MI455X will deliver 34x MI355X token throughput — Beth_Kindig · 2026-07-28
- Google Cloud adds near-real-time billing anomaly alerts for Gemini API and Vertex AI — rseroter · 2026-07-28
- LlamaIndex releases create-llama-worker to deploy LlamaParse on Cloudflare Workers — llama_index · 2026-07-28
- OpenRouter shows provider-level pricing, latency, and routing modes for the same model — gnukeith · 2026-07-28
- Kimi K3 serving uses relative GPU queue discounts to boost load balancing — Abhishekcur · 2026-07-28