Tsinghua-PKU NeurIPS Paper: Massive Activations Act as Gradient Regulators

机器之心 · wechat · 2026-11-21

A NeurIPS 2026 paper from Tsinghua and Peking University links Attention Sinks to Massive Activations via backpropagation: attention concentrates gradients on the first token ("Gradient Sinks", 1-2 orders of magnitude higher), and RMSNorm shrinks these gradients more when activations are larger. Their V-scale intervention cut the gradient-sink ratio from 40.3 to 5.6 in a 0.3B model while preserving attention sinks, supporting the causal story.

Original post →

More from Research

Research channel →