Study reveals cross-layer activation patterns in hybrid attention models

机器之心 · wechat · 2026-09-01

Researchers from StartLux, Tsinghua University, and others have systematically characterized the "cross-layer traces" left by sparse FullAttention layers in Hybrid Linear Attention LLMs (HLALLM) using a probe called "MassiveActivations."

Key Findings: Spikes and Plateaus

Pre-Attention Spikes (PAS): In the LinearAttention interval preceding a FullAttention layer, MassiveActivations (MAs) linked to the "Global Attention Sink" consistently peak at the layer immediately before the FullAttention. This pattern holds across five LinearAttention backbones (RetNet, HGRN, GLA, etc.) with an average occurrence rate exceeding 99.4%.
Inter-Spike Plateaus (ISP): As FullAttention layers become denser, the衰减 between isolated spikes diminishes. Instead, MAs remain high, forming a sustained plateau that connects adjacent spikes.

Mechanism: Write—Sink—Cancel

The study proposes a three-stage lifecycle to explain this phenomenon:

Write: The layer before FullAttention forms extreme values at specific coordinates.
Sink: Inside FullAttention, the token receives disproportionate attention.
Cancel: Opposite-sign updates cancel out the outliers.

Broad Validation

This pattern is verified in public models like KimiLinear, Qwen3.5, and Nemotron-H (1.2B-397B) and emerges early in training. The findings suggest that FullAttention layout acts as an "organizer" for the internal computational rhythm, offering new insights for model compression and quantization.

Original post →

Study reveals cross-layer activation patterns in hybrid attention models

More from Research