Weight normalization emerges as new standard: neuron norms hit almost exactly 1 in nGPT-style models
SonglinYang4 · x · 2026-11-13
wenkaiyue observes that in nGPT-style weight-normalized models the norm corresponding to each neuron is almost exactly 1, calling weight normalization the winner. Songlin Yang adds that weight normalization (nGPT, hyper ball, etc.) is becoming the new standard practice.
More from Research
- JoMoHa jointly RL-trains model and harness, hits 97.0% on ALFWorld — omarsar0 · 2026-11-13
- LATENTLENS: training-free method decodes what image tokens mean inside LLMs — burkov · 2026-11-13
- YC F26 team open-sources runtime to run 10,000 agents in one session — KlausCodes · 2026-11-13
- DynaDFlash: diffusion drafter boosts speculative decoding acceptance length by 5% — charles_irl · 2026-11-13
- Waterloo authors' textbook 'Elements of Dimensionality Reduction and Manifold Learning' has 392 citations — FrnkNlsn · 2026-11-13
- mcbench-lite open-sourced: agents build in Minecraft, runs locally on your keys — adonis_singh · 2026-11-13