Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong, Chuan-Wei Kuo
cs.CL
2026-08-12
In hybrid linear-attention LLMs, massive activations spike before full-attention layers and merge into plateaus as attention density rises, from 1.2B to 397B parameters.
Modern LLMs like Qwen3-Next, Qwen3.5, and Kimi Linear increasingly interleave linear-attention layers (cheap, fixed-size state) with full-attention layers (expensive, but expressive) to balance efficiency and capability. Nobody had systematically studied what this interleaving does to internal activation dynamics. Full-attention Transformers are known to develop massive activations (MAs): a handful of hidden-state entries at specific tokens that are orders of magnitude larger than typical values, tightly coupled to attention sinks (tokens that soak up a disproportionate share of attention regardless of content). This paper asks whether that same structure survives once linear attention gets mixed in, or whether hybridization reorganizes it.
The first obstacle is methodological. In full-attention models, the token with the largest activation and the attention-sink token are usually the same, so tracking largest magnitude is a reliable proxy for tracking the sink. In hybrid models that correspondence breaks down: the paper measures markedly weaker alignment between magnitude-ranked and sink tokens in a 1.3B hybrid, and the identity of the max-activation token flips between adjacent layers far more often than in a pure Transformer. The fix is to anchor on consensus attention sinks, sink tokens identified by averaging attention patterns across full-attention layers and heads, and track that fixed token's activation through depth, rather than re-finding the max-activation token at every layer.
Applying this across five linear-attention backbones (RetNet, HGRN, GLA, DeltaNet, GDN) and multiple hybridization ratios reveals two recurring patterns. Pre-attention spikes (PAS): the sink token's activation jumps sharply right before every full-attention layer. Inter-spike plateaus (ISP): when full-attention layers are packed more densely, the activation between consecutive PAS stops decaying and stays elevated instead. As full attention gets denser, ISP grows until PAS and ISP merge into the single, stable, elevated-everywhere MA pattern known from pure full-attention models.
Across the M-A-P suite (five architectures, 340M and 1.3B scales), the sink-to-spike alignment rate is near or at 100% almost everywhere:
| Architecture | 1.3B alignment | 340M alignment |
| HGRN | 100.0% | 100.0% |
| GDN | 100.0% | 100.0% |
| RetNet | 99.9% | 100.0% |
Inter-spike retention rises monotonically with full-attention density: for GDN at a 12:1 ratio (sparse full attention), the 1.3B model's retention is only 18.4%; at 3:1 (denser), it's 77.8%. Testing 12 public checkpoints from Kimi Linear, Qwen3.5, Nemotron-H, and Zamba2 (1.2B to 397B parameters) reproduces the same PAS/ISP structure, with matched Kimi Linear Base and Instruct checkpoints keeping nearly identical spike positions despite different magnitudes. Controlled pretraining shows adding output gates to full-attention layers strongly suppresses (but doesn't eliminate) PAS/ISP magnitude, while removing GDN's own gates only moderately amplifies them, suggesting full attention plays the dominant role in organizing this structure.
For anyone quantizing, pruning, or optimizing KV caches for hybrid models, this is a practical map of where the outliers live: not spread evenly across layers, but concentrated at specific tokens right before each full-attention layer, and the exact position shifts predictably with the attention schedule. Applying quantization heuristics tuned for pure Transformers uniformly across a hybrid model is likely to mishandle these pre-attention positions. The write-sink-cancel mechanism the paper proposes gives a concrete target for where to apply output gating or outlier-aware quantization: near full-attention layers, not evenly across the network.
The mechanistic account stays at the level of description and correlation; the paper is explicit that what regulates cancellation timing remains unresolved, and there's no lever offered for controlling the PAS-to-ISP transition directly. Controlled pretraining only reaches 1.3B parameters, while the open-source verification models go up to 397B, leaving a scale gap between the two lines of evidence: whether the early-training emergence of PAS/ISP observed at small scale still holds at large scale is untested. The paper also never connects these activation patterns to downstream capability: whether PAS or ISP matters for something like long-context recall is not tested at all; the entire study stays at the level of activation statistics.