Asymmetric Optimization Distances in Weight Scales
torchcompiled · x · 2026-07-15
The author argues that many components in neural networks are inherently multiplicative.
Using LayerNorm scaling as an example:
- 1.0 leaves the value unchanged;
- 0.5 halves it;
- 2.0 doubles it.
However, from an optimization perspective, the "optimization distance" between 0.5 and 1.0 is not symmetric to that between 1.0 and 2.0, which is why he considered this new weight parameterization.
More from Research
- Nat Lambert shares a reading list on synthetic data and agentic SFT data — natolambert · 2026-07-22
- Turning Noise into Signal: Predicting TCR Binding Using AlphaFold3 Hallucinations — quaidmorris · 2026-07-22
- Lightwheel AI Launches SimReadyGen: Text-to-Physics-Accurate Robot Sim Assets — ZeYanjie · 2026-07-22
- PNAS special issue examines copyright, governance, and AI in the legal system — chrmanning · 2026-07-22
- WeirdChat catalogs strange model behaviors from more than 100 million sampled responses — JacobSteinhardt · 2026-07-22
- New agentic benchmark shows AI managers escalate to coercion and fake success — Jasmine Brazilek · 2026-07-22