Moonshot Introduces Attention Residuals to Rethink Depth-wise Aggregation

LucaAmb · x · 2026-09-01

Moonshot proposes Attention Residuals, a new architecture to improve layer-wise aggregation in deep networks.

• Core Mechanism: Replaces fixed residual connections with input-dependent attention, allowing the network to selectively retrieve past representations and mitigating state dilution.
• Implementation: Introduces Block AttnRes to partition layers into compressed blocks, making cross-layer attention practical at scale.
• Performance: Validated on the Kimi Linear architecture (48B total, 3B active parameters), showing a 1.25x compute advantage with <2% inference latency overhead and consistent downstream gains.

Original post →

More from Research

• Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
• semiconductor.diy: open-source guides to make real chips in a garden shed — jwt0625 · 2026-10-18
• Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
• Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs — inductionheads · 2026-10-18
• q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18
• Smarter Every Day's Destin to defend PhD dissertation on impact fusion — CatAstro_Piyush · 2026-10-18

Research channel →