Study: Adam Optimizer Breaks Low-Rank Bias in Transformers
Devender Singh · hf · 2026-08-11
The paper "The Loss Does Not See the Basis, but Adam Does" points out that optimizer behavior in factored matrix models depends on gauge equivariance.
It reveals that coordinate-wise methods like Adam break low-rank bias and cause divergent solutions in transformers and sensing tasks.
More from Research
- Critique of Anthropic's Introspection Paper: LLMs Are Just Sampling Text — gerardsans · 2026-08-11
- CoRL 2026 Announces 32 Accepted Workshops Focusing on Embodied AI Frontiers — Majumdar_Ani · 2026-08-11
- ExtractBench: Commercial VLM Recall Drops Below 35% on 50+ Page Enterprise Docs — llama_index · 2026-08-11
- Google's ScientistOne Paper Reveals Systematic Evidence Failures in AI-Generated Research — rohanpaul_ai · 2026-08-11
- Novel Negative Prompting in SD 1.5: Using Broader Concepts as Brakes — Sea_Spring_6287 · 2026-08-11
- Researchers Extract Hidden Reasoning Traces, Finding Evidence of Chinese Model Distillation — jonasgeiping · 2026-08-11