Hera optimizer: fast global convergence for softmax attention via preconditioning
gautamcgoel · x · 2026-11-18
Sharing a peer's work, Gautam Goel (with Soltanolkotabi and Bartlett) points to his arXiv paper showing that training multilayer softmax self-attention on linear regression is equivalent to matrix factorization in the infinite-data limit. This motivates Hera, a first-order optimizer with a reparameterization and rescaling that approximates preconditioned updates. The Hera-PSD variant yields exponentially fast excess-risk decay, with bounds that don't degrade as attention depth grows.
More from Research
- Voyager 1 Reaches One Light-Day From Earth, Signals Now Take 24 Hours — snowclipsed · 2026-11-18
- TCS Hiring Statement Urges Quality Over Quantity as AI Fuels Paper Arms Race — fortnow · 2026-11-18
- New paper: a sociocultural approach to linguistic creativity — abenitezburraco · 2026-11-18
- 27B Model Rewrites Its Own Weights Without Training or Gradients, Paper Claims — udmrzn · 2026-11-18
- From 'impossible' to a working Bluetooth beacon: open-source EDA chip design is real — IgorCarron · 2026-11-18
- AURA 2.0: a streaming video LLM that speaks first, calls tools, and grounds visually — Shuhan Li · 2026-11-18