Foldable Weight Parameterization for Training Speedup
torchcompiled · x · 2026-07-15
Researchers propose a novel weight parameterization: representing weights as a learnable, weighted mixture of "linear weights + exponential weights." The author claims this approach can deliver up to 1.42x wall-clock speedup during training, and the weights can be folded back into standard weights for deployment afterward.
Further discussion notes that post-experimentation, the weight distribution was found to be more "heavy-tailed" than the baseline. Although the loss dropped, the author cautions that this might introduce other side effects requiring subsequent validation.
More from Research
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- VidMap uses RoMa coarse matching on all frames, fine-scale only for keyframes — ducha_aiki · 2026-09-11
- Bug Hunt Bench author: leaderboard noise is about 2-3 points — PawelHuryn · 2026-09-11
- PNAS paper shows a tiny billiard-ball system is a universal computer — undecidability lives in two dimensions — eigensteve · 2026-09-11
- New paper: Absolute pose estimation from affine cues and gravity direction — ducha_aiki · 2026-09-11
- LoMa Paper Ships REALLY HardPairs Dataset, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11