Asymmetric Optimization Distances in Weight Scales
torchcompiled · x · 2026-07-15
The author argues that many components in neural networks are inherently multiplicative.
Using LayerNorm scaling as an example:
- 1.0 leaves the value unchanged;
- 0.5 halves it;
- 2.0 doubles it.
However, from an optimization perspective, the "optimization distance" between 0.5 and 1.0 is not symmetric to that between 1.0 and 2.0, which is why he considered this new weight parameterization.
More from Research
- 3D ResNet Paper Crosses 3,000 Citations Eight Years After CVPR 2018 — HirokatuKataoka · 2026-09-11
- Sample selection and ordering matter a lot in LLM training: DataFlex makes data scheduling dynamic — Puzzleheaded_Box2842 · 2026-09-11
- Jeff Heaton's Intro to the Math of Neural Networks eBook Is Free to Download — blaizedsouza · 2026-09-11
- Mathematician Daniel Litt Launches Problem Repo to Track Human vs AI Progress: 15 Problems, 1 Solved — littmath · 2026-09-11
- Open ECDSA.fail challenge uses AI agents to shrink Shor's-algorithm quantum circuits for Bitcoin keys — StefanoGogioso · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11