Researchers weigh in on row norm: control pre-activation ∞ norm to tame RMS
CatAstro_Piyush · x · 2026-11-13
In a technical exchange on LLM normalization, YouJiacheng argues row norm (weight normalization) is the right approach: since SwiGLU involves elementwise multiplication, you must control the pre-activation hidden's ∞ norm to control the post-activation hidden's RMS. Cited experiments show each neuron's corresponding norm is almost exactly 1, supporting weight normalization.
More from Models
- Google's Carbon-Argon-Hydrogen model naming breaks its own size logic — JeremyNguyenPhD · 2026-11-13
- Microsoft and Claude go down simultaneously, taking apps offline with them — ThePeterMick · 2026-11-13
- Insider teases: Google still has a 'big juicy' model release up its sleeve — iruletheworldmo · 2026-11-13
- 1B model tops MathArena under-12B leaderboard with 83.62 — tensorqt · 2026-11-13
- Daily AI brief: Gemini 4 Hydrogen pricing revealed, Anthropic to list as ANT on Nasdaq Nov 19 — testingcatalog · 2026-11-13
- Unknown Darwin-180B-RSI posts perfect 100 on MathArena HMMT leaderboard, raising eyebrows — tensorqt · 2026-11-13