Hera optimizer: fast global convergence for softmax attention via preconditioning

gautamcgoel · x · 2026-11-18

Sharing a peer's work, Gautam Goel (with Soltanolkotabi and Bartlett) points to his arXiv paper showing that training multilayer softmax self-attention on linear regression is equivalent to matrix factorization in the infinite-data limit. This motivates Hera, a first-order optimizer with a reparameterization and rescaling that approximates preconditioned updates. The Hera-PSD variant yields exponentially fast excess-risk decay, with bounds that don't degrade as attention depth grows.

Original post →

More from Research

Research channel →