Transformer Massive Activations and Attention Sinks Share Same Root
alfcnz · x · 2026-07-09
This thread discusses two common phenomena in Transformers: massive activations in a few residual channels, and abnormal attention concentration on a specific "sink" token. The author claims that an ICML 2026 paper proves both stem from the same mechanism: early FFNs inject massive spikes that subsequent layers later cancel out. RMSNorm then forces the spiked token into an approximately fixed direction, making it a sink token that all queries easily attend to. The thread also explains the practical implications: massive activations break quantization because outliers shrink normal values, while attention sinks are useful for streaming inference, as fixing these sink tokens in the KV cache allows the model to run longer.
More from Research
- Baseten study finds new facts in LLM weights are fragile unless trained from many restatements — alex_verem · 2026-07-21
- Kimi K3 and Fable 5 now look much closer than the old open-vs-closed gap — FinanceYF5 · 2026-07-21
- uv-scripts/ocr returns to the top of Hugging Face datasets with a JSON model picker — vanstriendaniel · 2026-07-21
- DeepSearch-World trains web agents with 420K verifiable QA tasks — HKUST · 2026-07-21
- GigaAM Multilingual targets low-resource Central Asian ASR with 2M hours of audio — ai-sage · 2026-07-21
- WorldCupArena benchmarks language models on 104 football matches — Zhaokai Wang · 2026-07-21