Optimizer combining PSGD and KL-Shampoo avoids eig/inverse/solve
YouJiacheng · x · 2026-08-31
The author proposes an optimizer method combining PSGD-Kron's multiplicative update with the KL-Shampoo objective.
- Method: Uses the KL divergence $KL(N(0, \Sigma), N(0, P^{-2}))$ as the objective for the multiplicative update $P$.
- Advantage: It avoids eigen-decomposition, matrix inversion, or solving linear systems while being exact with gauge freedom handled.
- Insight: Notes that the PSGD update is effectively the relative gradient of KL-Shampoo.
Related event: PSGD Objective Found Mathematically Equivalent to KL-Shampoo(6 posts)→
More from Research
- Interpretability debate: one interpretable polysemantic neuron doesn't prove all are — aran_nayebi · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- SERO augments long-horizon agentic RL with executable oracle continuations — Shanghai-AI-Laboratory · 2026-10-16
- Tsinghua and Amap propose OmniVerifier-C1, a graded verifier beating binary judges — Bowei Liu · 2026-10-16
- Alibaba open-sources MAI-Gym, an all-in-one GUI agent eval and RL training platform — Tongyi-MAI · 2026-10-16