Why RMSNorm rarely blows up gradients near zero: high-dim init keeps vectors far away
carlesgelada · x · 2026-11-07
François Fleuret asked: given that an input near zero makes RMSNorm blow up gradients, why isn't it more often an issue? Carles Gelada's answer: a high-dimensional random vector is always initialized far away from 0, and under learning dynamics 0 is not an attractor — training doesn't pull parameters into states where inputs approach zero.
A short but substantive architecture exchange.
Related event: RMSNorm Gradient Explosion Near Zero: Question Raised and Explained(6 posts)→
More from Research
- METR: Anthropic researchers claim AI is speeding them up by 12x — we_are_mammals · 2026-11-07
- Anthropic uses Claude to map first 3D structure of bacterial vault particles — DeryaTR_ · 2026-11-07
- ART-Glove tactile capture glove goes open source, demo at CoRL — micoolcho · 2026-11-07
- Data centers' water use is just ~4% of US golf course irrigation, study argues — kimmonismus · 2026-11-07
- Nine AI agents tackle national lab open questions; none pass expert review — The Decoder · 2026-11-07
- Researcher Alleges Reviewers Use LLMs to Write Batch Rejections and Eliminate Rival Papers — examachine · 2026-11-07