Why RMSNorm rarely suffers exploding gradients near zero

François Fleuret questioned why RMSNorm rarely suffers exploding gradients when inputs approach zero. Carles Gelada explained that high-dimensional randomly initialized vectors almost always stay far from zero, making the issue rare in practice.

2026-11-07 ~ 2026-11-07 · 2 related posts

• François Fleuret asks why RMSNorm's near-zero gradient blowup rarely causes problems in practice — francoisfleuret · 2026-11-07
• Why RMSNorm rarely blows up gradients near zero: high-dim init keeps vectors far away — carlesgelada · 2026-11-07