Learned optimizers without 4,000 TPU months: new work claims general-purpose trained optimizers are finally practical
burkov · x · 2026-10-13
Modern foundation model training relies on hand-designed optimizers like Adam, while learned optimizers—whose update rules are themselves trained on data—have stalled due to poor generalization, instability at scale, and astronomical costs: prior SOTA VeLO consumed 4,000 TPU months yet failed to reliably train models beyond 600M parameters.
This article argues an effective, general-purpose learned optimizer doesn't require enormous training compute, presenting methods and results that could replace a hand-crafted step in training and cut both training time and cost.
More from Infra
- AMD Ramps Helios Rack Into Production a Year After Debut, Unveils Helion Rack and Verano CPU — ryanshrout · 2026-10-13
- Can you mix an RTX 5060 Ti with a 7900XTX for LLM work? CUDA + ROCm together — hedgehog0 · 2026-10-13
- n+2 lookahead expert prefetching for Qwen Flash Next SSD streaming nets +8% on Mac — turtleninja99 · 2026-10-13
- Muse Wonder requests die with 502 exactly 300s after last output on OpenRouter — adonis_singh · 2026-10-13
- SonicBoom: A Python-Free C++ Runtime Beats llama.cpp Decode by 14% on a Laptop 3050 — nalaginrut · 2026-10-13
- GP builds local clinical assistant on one RTX 4090: Gemma 4 26B QAT cuts response to ~3 seconds — looneycheetah · 2026-10-13