M+Adam Improves Low-Precision Training
AnimaAnandkumar · x · 2026-07-14
Recent findings show that for transformer models with **under 1B parameters** and training budgets ranging from **1x to 8x Chinchilla**, **M+Adam** consistently outperforms **AdamW** in low-precision master-weight settings. Key conclusions: - The benefits are most significant in **more aggressive low-precision training** scenarios. - Low-precision training requires the optimizer's update geometry to align with the floating-point grid. - M+Adam combines scale-aware progression and local correction by performing both **multiplicative updates** and **additive updates** within a single optimization step. - This method supports **end-to-end low-precision training**, eliminating the reliance on higher-precision master weights.
Related event: M+Adam Optimizer Improves Low-Precision Training(2 posts)→
More from Research
- Draft paper uses Markov-chain eigenfunctions to build partitions and speed up sampling — michaelchchoi · 2026-07-21
- Autoresearch proposes packaging ML runs as studies with questions, analysis, and code diffs — morgymcg · 2026-07-21
- GitHub repo adds lightweight ternary QAT for Prism-ML Bonsai models — terminoid_ · 2026-07-21
- Qdrant co-hosts a Munich meetup on search, retrieval, and agentic RAG on July 23 — qdrant_engine · 2026-07-21
- GigaChat Audio targets long-form audio grounding with timestamps across 120-minute inputs — ai-sage · 2026-07-21
- Paper models Transformer components as stochastic geometry and tests five architectures — Zhihua Liang · 2026-07-21