Weight decay causally controls the life cycle of massive activations, paper shows
Aimpoint-Digital · hf · 2026-10-12
A paper from Aimpoint Digital Labs, Mistral AI and NYU traces the full life cycle of massive activations — residual-stream outliers linked to attention sinks.
- Sink-carrying channels vary across random seeds but stabilize early within each run; over training, surrounding channels erode and the sink consolidates onto a few redundant carriers.
- Central result: weight decay causally regulates global activation scale — removing it near the peak lets the scale keep rising, while retaining it causes decline even at a constant learning rate.
- They build a balance model where AdamW-preconditioned growth opposes weight decay: sweeping decay coefficient λ shifts peak timing roughly log-linearly, with peak magnitude scaling approximately as λ^{-1/2}.
- Gradient attenuation follows the sink token's collective RMS magnitude rather than any single channel.
The work establishes weight decay as a training-time lever on activation magnitude. Code is open-sourced.
More from Research
- Critique says OpenAI's Navier-Stokes Lean formalization checked a different argument — thione · 2026-10-12
- OpenAI drops 700+ math manuscripts; human mathematicians group urges boycott — thione · 2026-10-12
- Biohub expands AI-biology data effort to $1.8B with US agencies, Meta and Google — thione · 2026-10-12
- Paper: DMD turns diffusion models into looped models iterating in data space — CSProfKGD · 2026-10-12
- Jina AI to host 4th BoF on embeddings, rerankers & small LMs at EMNLP 2026 — JinaAI_ · 2026-10-12
- Yoav Goldberg defends OpenAI's FFT result: critique TCS culture, not the paper — yoavgo · 2026-10-12