Tsinghua-PKU NeurIPS Paper: Massive Activations Act as Gradient Regulators
机器之心 · wechat · 2026-11-21
A NeurIPS 2026 paper from Tsinghua and Peking University links Attention Sinks to Massive Activations via backpropagation: attention concentrates gradients on the first token ("Gradient Sinks", 1-2 orders of magnitude higher), and RMSNorm shrinks these gradients more when activations are larger. Their V-scale intervention cut the gradient-sink ratio from 40.3 to 5.6 in a 0.3B model while preserving attention sinks, supporting the causal story.
More from Research
- Horvitz and Mitchell Publish 'Scientific Progress in AI: History, Status, and Futures' — ArtificialOther · 2026-11-22
- Interview: Medical AI is only as good as human consensus labels — CurieuxExplorer · 2026-11-22
- Science cover paper runs contextuality tasks on up to 105 qubits of Google's Willow chip — dhruv2038 · 2026-11-22
- Why a P=NP proof would collapse most of modern cryptography — doctorBobG · 2026-11-22
- 10 months after a Reddit question on AI solving Millennium Prize Problems, claims abound — haider1 · 2026-11-22
- Fable 5.2 rewrites model weights so an E2E driving bot learns left-side driving — 4310sy · 2026-11-22