Why Adam Underperforms GD: Paper Reveals Symmetry Breaking Causes 40% Higher Error

burny_tech · x · 2026-08-09

A new arXiv preprint explores why Adam and Standard Gradient Descent (GD) behave entirely differently on the same loss landscape.

• Core Insight: GD naturally respects gauge symmetry in factored models, smoothly picking up low-rank solutions. In contrast, Adam and other coordinate-wise optimizers shatter this symmetry instantly by scaling gradients entry by entry.
• Transformers Impact: In Transformers, Adam splits Gauge Equivalent Initializations on the very first step, leaving a 56% gap in per-head invariants that cannot be patched later.
• Performance: On underdetermined matrix tasks, GD cuts test error by over 40% compared to adaptive methods purely due to symmetry preservation.

Original post →

More from Research

• MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
• DIY LEGO bench tests AI brick-building, GPT-6 Astra beats Opus — TheLegoless · 2026-10-16
• tszzl's theory: Tornado → Apache license → apaches → Southwest → Colorado — tszzl · 2026-10-16
• DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
• A Qwen3 neuron tracks both Colorado and Tornado HTTP code; tszzl offers a theory — tszzl · 2026-10-16
• AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16

Research channel →