Asymmetric Optimization Distances in Weight Scales
torchcompiled · x · 2026-07-15
The author argues that many components in neural networks are inherently multiplicative.
Using LayerNorm scaling as an example:
- 1.0 leaves the value unchanged;
- 0.5 halves it;
- 2.0 doubles it.
However, from an optimization perspective, the "optimization distance" between 0.5 and 1.0 is not symmetric to that between 1.0 and 2.0, which is why he considered this new weight parameterization.
More from Research
- RSNA 2026 Deep Learning Labs return: eight hands-on AI classes over four days in Chicago — PRouzrokh · 2026-10-27
- Paper: data schedules and learning rate schedules should be designed together — alifcoder · 2026-10-27
- Health benchmark study: quality and diversity of eval items beat quantity — keunwoochoi · 2026-10-27
- Across 7 tasks, highest reasoning setting isn't the best for most open models — keunwoochoi · 2026-10-27
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open Models Match GPT-6 on MedMCQA but Trail 17 Points on Hard Clinical Cases — keunwoochoi · 2026-10-27