Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers
Yihong Chen, Zhouchen Lin, Quanming Yao
cs.LG, cs.AI
2026-03-18
Attention sinks pool gradients on token 0; massive activations relieve pressure via RMSNorm. A value-path gradient valve suppresses MA and adds up to 15 points on retrieval.
Attention sinks and massive activations travel together in LLMs: under causal masking the first token absorbs a large share of attention mass, and a few token-feature positions carry activations orders of magnitude larger than everything else. Both complicate long-context inference and fine-tuning, and quantization most of all, since a handful of extreme values stretches the range that low-bit formats must cover.
Prior explanations live mostly on the forward pass: sinks as "no-op" heads parked under softmax's sum-to-one constraint, massive activations as something that helps sustain the sink. That account hits a wall with pre-norm architectures. Attention and MLP sublayers consume RMSNorm-normalized inputs, so the raw norm of the residual stream barely affects the forward computation, and nothing forward-facing explains why training pushes sink-token activations to such extremes. This NeurIPS 2026 paper from Tsinghua and Peking University moves the question to the backward pass and argues that massive activations are gradient pressure valves the model builds for itself.
The core observation fits in one line: attention weights are two-way routers. Forward, they aggregate values into later positions; backward, the same weights carry the gradients of every later position back into the one being attended. Under a causal mask, a long run of future tokens attends to the first token, so their value-path gradients stack up there. The paper calls this concentration a gradient sink.
The chain is nailed down in three steps:
The intervention that tests this is V-scale. After the value projection, v becomes φ(‖v‖²)v with φ(r) = r/(r+C), one learned parameter per layer and head. Sink tokens are known to carry small value norms, so when r is far below C the backward Jacobian's eigenvalues are on the order of r/C and the sink gradient is strongly attenuated, while large-norm values pass through nearly unchanged. A backward-only control keeps the forward values bit-identical (stop-gradient splice) and swaps only the backward rule, separating "changed gradients" from "changed forward values."
| Comparison | Metric | Result |
| 0.3B baseline → V-scale | token-0 gradient concentration (median) | 40.3x → 5.6x |
| 1B baseline vs V-scale | Wiki perplexity | 22.84 vs 22.83 |
| Multi-key NIAH, bfloat16, YaRN 8192 | retrieval accuracy | 48.32% → 62.00% |
| Multi-key NIAH, 8192, all precisions | vs baseline | +8.68 to +15.04 points |
| Static per-tensor W8A8, 2048 | baseline retrieval | near zero; V-scale keeps most of it |
Mechanistically, V-scale preserves attention sinks at both scales (slightly stronger in some layers) while cutting token-0 norms across the residual stream, attention output, and MLP output. The largest drop is in MLP outputs, which V-scale never touches on the forward side; this is the strongest causal evidence that gradient pressure drives massive activations. The backward-only control shows the same pattern, so changing the backward rule alone is enough.
On standard benchmarks the 1B pair is comparable overall, Wiki perplexity 22.84 against 22.83, with gains on ARC-C and BoolQ and small moves elsewhere. Multi-key NIAH (the many-keys variant of Needle-in-a-Haystack, where several competing key-value pairs are buried in a long context) favors V-scale in every tested precision-by-length combination, including YaRN extension to 8192. The static per-tensor gap is the most dramatic: baseline retrieval at 2048 collapses toward zero while V-scale keeps most of its capability, and quantizing only the MLP downproj reproduces the contrast.
For training, the paper flips the instinct that massive activations are defects to clamp. They behave like pressure-relief valves the model grows on its own. Activation-smoothing and outlier-suppression schemes that touch activations without touching the gradient pressure may just displace it; removing MA cleanly means opening another pressure route first, which is exactly what V-scale does.
For deployment the gains are close to free: suppressing MA markedly improves tolerance of coarse activation quantization (static per-tensor W8A8 retrieval goes from collapsed to mostly intact) and steadies quantized multi-key retrieval, at one parameter per layer and head.
For interpretability, it turns the "are AS and MA the same thing" debate into an operational question: control the gradient sink and the two come apart, sinks stay, activations go.
The authors state the scope: Llama-like dense language models only, with MoE and multimodal Transformers untested, and other factors such as data distribution, optimizer, and numerical precision not excluded as shapers of AS and MA.
Discounts worth applying: the intervention runs top out at 1B, while the 7B-9B open models only get gradient diagnostics, so the effect at larger scale is untested. Harder single-needle tasks degrade in some quantization-plus-YaRN combinations, and the paper concedes the cause is unresolved. Standard benchmarks are a wash rather than a win, and the downstream gains concentrate in competitive multi-key long-context retrieval.