Kimi Introduces Attention Residuals Architecture with <2% Inference Overhead

ChengleiSi · x · 2026-08-13

Moonshot AI (Kimi) introduced Attention Residuals, a novel architecture rethinking depth-wise aggregation in neural networks.

Core Innovation: It replaces standard, fixed depth-wise recurrence with learned, input-dependent attention over preceding layers. This allows the network to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth.
Block AttnRes: To make cross-layer attention practical at scale, the method partitions layers into compressed blocks.
Performance: Serving as an efficient drop-in replacement, it demonstrates a 1.25x compute advantage with negligible (<2%) inference latency overhead.

Furthermore, researchers highlighted the architectural elegance of this design: it naturally preserves norms in the forward pass via softmax and provides excellent gradient shortcuts in the backward pass, making the optimization process highly principled.

Original post →

Kimi Introduces Attention Residuals Architecture with <2% Inference Overhead

More from Research