Orabona: the classic preconditioning story for full-matrix AdaGrad doesn't hold up
bremen79 · x · 2026-11-18
Optimization theorist Francesco Orabona published a new post challenging the deep learning mythology around preconditioning.
- Classical definition vs DL myth: in classical optimization, preconditioning has a precise role — improving the Hessian's condition number to speed convergence; the DL story of "matrices adapt to geometry" lacks that rigor
- TL;DR: the full-matrix AdaGrad bound is never better, and often worse, than the scalar-learning-rate bound — the same reasoning yields a better bound with the matrices removed, undermining the classic justification
- "Alchemy" critique: good algorithms don't need theory to work, but insisted-upon explanations must make sense; the sin is motivating heuristics with nonsensical theory, like alchemists whose reactions were real but explanations weren't
- What a proper theory needs: it must identify what improves (condition number, curvature–noise relation, a real geometry) and verify assumptions in the real world
Directly relevant to the theoretical foundations of matrix-style optimizers like Shampoo.
More from Research
- TCS Hiring Statement Urges Quality Over Quantity as AI Fuels Paper Arms Race — fortnow · 2026-11-18
- New paper: a sociocultural approach to linguistic creativity — abenitezburraco · 2026-11-18
- 27B Model Rewrites Its Own Weights Without Training or Gradients, Paper Claims — udmrzn · 2026-11-18
- From 'impossible' to a working Bluetooth beacon: open-source EDA chip design is real — IgorCarron · 2026-11-18
- AURA 2.0: a streaming video LLM that speaks first, calls tools, and grounds visually — Shuhan Li · 2026-11-18
- StellarEdge Auriga report: verifiable execution harness hits 57.74% on 42 RoboDojo tasks — jiqizhixin · 2026-11-18