OKLS brings KL-optimal Shampoo to language model training with 1.45× parameter efficiency
aryaman2020 · x · 2026-07-29
OKLS proposes a KL-optimal Shampoo variant for language model training
The paper introduces Online KL Shampoo (OKLS), an optimizer that approximates full-matrix AdaGrad more closely than diagonal methods, while staying practical for large-scale language model training.
- It uses KL-optimal Kronecker factors to whiten gradients across both row and column dimensions, aiming to capture parameter correlations that diagonal optimizers miss.
- The authors argue it improves on Muon, which captures correlations but not historical information, by making the preconditioner more faithful to full-matrix geometry.
- A key engineering challenge is keeping the inverse-square-root preconditioner fresh at every step; the paper proposes Scaled CANS Coupled Newton–Schulz to make zero-staleness preconditioning feasible.
- The implementation uses 10 iterations, 27 FP16 GEMMs, and FP32 accumulation.
- Reported results: 1.45× the parameter efficiency of Muon while retaining 98% of training throughput, across 200M–1B scale models.
More from Research
- LLMs on robots jump real-world success from 16.7% to 97.3% — tri_dao · 2026-07-29
- Simple teleoperation recordings plus soft compliance can solve more robot tasks than expected — ihorbeaver · 2026-07-29
- Studies find AI therapy replies often score higher on empathy than human clinicians — sapinker · 2026-07-29
- TransluceAI proposes oversight foundation models to catch reward hacking at scale — JacobSteinhardt · 2026-07-29
- Student builds a local coding agent on molab and beats two 7B coder baselines — S_Conradi · 2026-07-29
- Why tokenizer optimization is hard: expensive pretraining, slow feedback, and non-differentiable design — paul_cal · 2026-07-29