Moonshot Introduces Attention Residuals to Rethink Depth-wise Aggregation

LucaAmb · x · 2026-09-01

Moonshot proposes Attention Residuals, a new architecture to improve layer-wise aggregation in deep networks.

• Core Mechanism: Replaces fixed residual connections with input-dependent attention, allowing the network to selectively retrieve past representations and mitigating state dilution.
• Implementation: Introduces Block AttnRes to partition layers into compressed blocks, making cross-layer attention practical at scale.
• Performance: Validated on the Kimi Linear architecture (48B total, 3B active parameters), showing a 1.25x compute advantage with <2% inference latency overhead and consistent downstream gains.

Original post →

More from Research

• ECCV 2026 AI Art Gallery launches online with 52 accepted works — CSProfKGD · 2026-09-01
• Embodiment-aware RL for robot control presented at three international workshops — Jan_R_Peters · 2026-09-01
• Aardvark Weather Model Gains Probabilistic Capabilities, Separating Observation and Model Uncertainty — FraunhoferHHI · 2026-09-01
• NBER Paper: Best LLM for Automation Often Lags at Assisting Weaker Models — soumitrashukla9 · 2026-09-01
• Xiaohongshu & NVIDIA build GR-Inference engine, doubling throughput for Beam Search — 小红书技术REDtech · 2026-09-01
• Frontier models depreciate rapidly as Nvidia revenue doubles to $96.2B — Exponential View (Azeem Azhar) · 2026-09-01

Research channel →