Pre-norm breaks scale invariance: do modern Transformers invalidate a core DL theory assumption?
aaron_defazio · x · 2026-11-10
- @miyashita03 argues many deep learning theory papers assume scale invariance f(cw)=f(w) arises from normalization layers, but modern Transformers may not satisfy it.
- Pre-norm (standard in today's LLMs) doesn't make even a simple linear layer scale-invariant; SwiGLU and attention break it too, and QK-norm only covers Q/K projection weights.
- Yet the assumption empirically underpins optimizers like AdamC, AdamMD and AdamH — raising the question of where scale invariance actually comes from. The author plans to read Jianlin Su's article on the topic.
- The attached figure compares pre-norm, post-norm and sandwich-norm under weight scaling by c: only sandwich-norm keeps y(cW)=y(W).
More from Research
- Newton: An open-source GPU-accelerated physics engine built on NVIDIA Warp — ChongZzZhang · 2026-11-11
- From quantum to AI: Ge Yang on CoRL's boom and where robot learning goes next — EpisodeYang · 2026-11-11
- Newton-FBA ships sim2real cloth manipulation, learned in minutes — ChongZzZhang · 2026-11-11
- Stanford AI100 releases third report: 'AI Goes Mainstream' on the scaling era — StanfordHAI · 2026-11-11
- Video explainer: sleep-time compute and offline memory consolidation as LLM's next scaling axis — bycloud · 2026-11-11
- 26,040 agent runs and 100B tokens: FrontierHarness v1.1 ranks harnesses across 5 models — stuffyokodraws · 2026-11-11