A Deep Derivation of Transformer Init and muP: Why Learning Rates Must Scale With Width
青稞AI · wechat · 2026-09-19
A rigorous long-form article from the Qingke AI community derives the math behind Transformer initialization and extends it to width-scaling learning rates.
Key threads:
- LeCun/Xavier/Kaiming init: derivations from forward second-moment preservation, the backward-pass tradeoff, and ReLU-style nonlinearities, including Xavier's forward/backward compromise for non-square matrices.
- Residuals & normalization: why residual stream variance accumulates with depth and where nanoGPT/GPT-2's 0.02 init and cproj scaling come from.
- muP parametrization: how muP jointly sets init variance, forward multipliers, and per-layer learning-rate exponents to keep feature changes constant-order across widths; why the output layer needs smaller init variance.
- Optimizers: learning-rate scaling derived for SGD, Adam and Muon, unified via a spectral-norm condition on weights and updates.
Aimed at readers who want to deeply understand training stability and the principles behind muP.
More from Research
- Unified 3D generative model for synthesizable structure-based drug design released as preprint — charlieharris01 · 2026-09-20
- ICLR 2027 hits 60,000+ submissions, nearly 90% possibly AI-written — DominiqueCAPaul · 2026-09-20
- SVE2 match instructions speed up JSON parsing in simd on ARM — lemire · 2026-09-20
- Remote Labor Index adds Fable and Astra, now spanning 6,000+ hours of real $140K professional work — Alex__007 · 2026-09-20
- Reconstructing Dynamics, Not Values: A Delay-Embedding Approach to Gappy Time Series — bravo_abad · 2026-09-20
- Author of 'LLMs feel pain' study tells Gary Marcus: we never claimed that — GaryMarcus · 2026-09-20