SEL Weight Transformation Achieves 1.42x Speedup
torchcompiled · x · 2026-07-15
This post introduces the author's proposed SymExpLin (SEL) weight transformation, claiming up to a 1.42x wall-clock speedup during training.
The core idea is:
- The SymExp component converts additive updates into multiplicative ones, introducing learnable curvature parameters;
- It retains a standard linear path, allowing the network to learn how to weight and fuse the two paths.
The author also notes that this transformation can ultimately be fused back into standard weights for deployment.
More from Research
- 3D ResNet Paper Crosses 3,000 Citations Eight Years After CVPR 2018 — HirokatuKataoka · 2026-09-11
- Sample selection and ordering matter a lot in LLM training: DataFlex makes data scheduling dynamic — Puzzleheaded_Box2842 · 2026-09-11
- Jeff Heaton's Intro to the Math of Neural Networks eBook Is Free to Download — blaizedsouza · 2026-09-11
- Mathematician Daniel Litt Launches Problem Repo to Track Human vs AI Progress: 15 Problems, 1 Solved — littmath · 2026-09-11
- Open ECDSA.fail challenge uses AI agents to shrink Shor's-algorithm quantum circuits for Bitcoin keys — StefanoGogioso · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11