Optimizer combining PSGD and KL-Shampoo avoids eig/inverse/solve
YouJiacheng · x · 2026-08-31
The author proposes an optimizer method combining PSGD-Kron's multiplicative update with the KL-Shampoo objective.
- Method: Uses the KL divergence $KL(N(0, \Sigma), N(0, P^{-2}))$ as the objective for the multiplicative update $P$.
- Advantage: It avoids eigen-decomposition, matrix inversion, or solving linear systems while being exact with gauge freedom handled.
- Insight: Notes that the PSGD update is effectively the relative gradient of KL-Shampoo.
More from Research
- NeurReps 2026 CFP: Symmetry and Geometry in Neural Representations — fatihdin4en · 2026-09-01
- Dan Luu on why software slowness is a choice, analyzing latency costs and optimization — JeremyCMorgan · 2026-09-01
- Scholar calls out LLM gibberish: reviewing papers and replies is now a waste of time — thegautamkamath · 2026-09-01
- Paper analyzes reasoning models like o1 and DeepSeek R1, probing CoT data contamination — rao2z · 2026-09-01
- Qdrant's Sept 17 stream: token-native storage claims 10-100x faster reads — qdrant_engine · 2026-09-01
- Qdrant Event Preview: Benchmarks on Hybrid Search Tuning Parameters — qdrant_engine · 2026-09-01