Study reveals cross-layer activation patterns in hybrid attention models
机器之心 · wechat · 2026-09-01
Researchers from StartLux, Tsinghua University, and others have systematically characterized the "cross-layer traces" left by sparse FullAttention layers in Hybrid Linear Attention LLMs (HLALLM) using a probe called "MassiveActivations."
Key Findings: Spikes and Plateaus
- Pre-Attention Spikes (PAS): In the LinearAttention interval preceding a FullAttention layer, MassiveActivations (MAs) linked to the "Global Attention Sink" consistently peak at the layer immediately before the FullAttention. This pattern holds across five LinearAttention backbones (RetNet, HGRN, GLA, etc.) with an average occurrence rate exceeding 99.4%.
- Inter-Spike Plateaus (ISP): As FullAttention layers become denser, the衰减 between isolated spikes diminishes. Instead, MAs remain high, forming a sustained plateau that connects adjacent spikes.
Mechanism: Write—Sink—Cancel
The study proposes a three-stage lifecycle to explain this phenomenon:
- Write: The layer before FullAttention forms extreme values at specific coordinates.
- Sink: Inside FullAttention, the token receives disproportionate attention.
- Cancel: Opposite-sign updates cancel out the outliers.
Broad Validation
This pattern is verified in public models like KimiLinear, Qwen3.5, and Nemotron-H (1.2B-397B) and emerges early in training. The findings suggest that FullAttention layout acts as an "organizer" for the internal computational rhythm, offering new insights for model compression and quantization.
More from Research
- Stanford's Zitnik: Biomedical Discovery Lacks Hard Verifiers, Blocking AI Scientist Loops — marinkazitnik · 2026-09-01
- Closing the Loop: The Rise of Autonomous AI Scientists — marinkazitnik · 2026-09-01
- Tencent Hunyuan AngelSlim: Compressing Hy4 Model to 214GB with Heterogeneous Inference — 腾讯混元 · 2026-09-01
- AlphaEvolve sets new record for matrix multiplication exponent: omega to 2.371177 — 新智元 · 2026-09-01
- Building a long-term memory benchmark for agents: what to add? — True_Mongoose_7073 · 2026-09-01
- PaperGym: Rubric-centered evolution for research-plan generation — Yuhan Wang · 2026-09-01