Kimi Team proposes Attention Residuals, replacing fixed residual accumulation with learned attention
burny_tech · x · 2026-10-27
Kimi Team's new paper Attention Residuals argues that standard PreNorm residual connections aggregate all layer outputs with fixed unit weights, causing uncontrolled hidden-state growth that dilutes each layer's contribution. AttnRes instead lets each layer selectively aggregate earlier representations via softmax attention with learned, input-dependent weights. To cut memory/communication overhead at scale, Block AttnRes partitions layers into blocks and attends over block-level representations, working as a practical drop-in replacement. Scaling-law runs show consistent gains across sizes. Developer burnytech reports adding AttnRes to the General Physics Transformer: a reproduced table shows Block AttnRes cutting radiative-layer 12-step NMSE from 0.39277 to 0.28917, with faster loss convergence.
More from Models
- Sakana AI Announces Pretraining Its Own LLM, Launches 'Sakana Sawara' Project — SakanaAILabs · 2026-10-27
- Nemotron-3.5 Bolt hits 9B active params, within 1% of Super on agentic benchmark — ryanshrout · 2026-10-27
- Cohere Accused of Spreading FUD About Chinese Models — voooooogel · 2026-10-27
- Steering two neurons turns Qwen3.5 4B into "evil qwen" — here's the math — burny_tech · 2026-10-27
- Opus 3's origin story survives repeated context compactions, seam after seam — repligate · 2026-10-27
- Human-written essay flagged as Mixed by AI detector Pangram in blogger's test — Aizkmusic · 2026-10-27