SEL Weight Reparameterization: ~1.42x Training Speedup, Folds Back to Standard Weights

On July 15, @torchcompiled published a series of posts introducing his weight parameterization method SymExpLin (SEL). Its core pitch: during training, weights are represented as a learnable weighted blend of SymExp and ordinary linear weights, reportedly yielding up to ~1.42x wall-clock training speedup; after training they can be folded back into standard weights, adding no deployment burden. This touches both training efficiency and deployment compatibility at once, and the author calls it his most interesting current project.

Key Mechanism

According to the posts, SEL has two components. The first is SymExp: it converts the traditional additive update into a more multiplicative form and introduces a learnable curvature parameter. The second is a learnable blend with ordinary linear weights, rather than fully replacing the original parameterization. The author also notes the design includes magnitude-dependent scaling, intended to make both the relative and absolute movement of parameters more sensible across scales.

Design Motivation

The author recalls first trying to intervene at the optimizer level, but that tended to diverge when changing per-dimension step sizes and preconditioning simultaneously, so the idea shifted to “keep the preconditioner fixed and transform the weights themselves.” He also argues that much of a neural network is inherently multiplicative: for a LayerNorm scale, 1.0 is no change, 0.5 roughly halves and 2.0 roughly doubles, yet the two are not symmetric in terms of “distance” under ordinary optimization. Another starting point is that pretrained model weights are often heavy-tailed, and existing preconditioning does not truly capture the optimization distance some parameters must traverse.

Current Conclusion

The posts position SEL as a foldable, deployable training-speedup technique, with the author claiming up to ~1.42x wall-clock speedup. However, the posts mainly lay out the design rationale and effect overview, without elaborating on experimental setup, applicable model scope, or independent reproduction in the available material.

2026-07-15 ~ 2026-07-15 · 7 related posts

• [source] Foldable Weight Parameterization for Training Speedup — torchcompiled · 2026-07-15
• [source] Learnable Weight Mixing Accelerates Training — torchcompiled · 2026-07-15
• Pretrained Weights Exhibit Heavy-Tailed Distribution — torchcompiled · 2026-07-15
• Asymmetric Optimization Distances in Weight Scales — torchcompiled · 2026-07-15
• Shifting Focus from Optimizers to Weight Transformation — torchcompiled · 2026-07-15
• [source] SEL Weight Transformation Achieves 1.42x Speedup — torchcompiled · 2026-07-15
• Training Acceleration Trick Deployable via Weight Fusion — torchcompiled · 2026-07-15