arXiv paper: switch optimizers mid-training, save 65%+ of compute
RichmanRonald · x · 2026-08-23
The arXiv paper "Many Optimizers But Only One Training Path" (2608.18810, Ronald Richman and Mario Vüthrich) proposes Repeated Optimizer Resampling (ROR): instead of fixing an optimizer before training, choose it dynamically within a single evolving run.
Method: every $b$ epochs, each candidate optimizer "scouts" from the current weights for $s$ epochs; the best scout continues for the remaining $b-s$ epochs, and a completed segment becomes the new incumbent if the validation objective improves — allowing the preferred optimizer to change as training progresses.
On MNIST, Fashion-MNIST, and two motor insurance claim-count models, with identical seeds comparing 9 fixed optimizers and 2 ROR variants: one-epoch ROR uses only 24%-35% of the aggregate training needed to identify the best fixed optimizer exhaustively while staying close to it on all four tasks.
More from Research
- Depth Anything V4 Paper Withdrawn Over False Claims — JFPuget · 2026-08-23
- Review Paper: From Vector Autoregressions to AI-based Time Series Forecasting — PtrPomorski · 2026-08-23
- Study: AI Coding Agents Fail Coordination at Scale — rohanpaul_ai · 2026-08-23
- AI-Designed Dog Cancer Vaccine Startup Gamgee Raises $4M Seed — 机器之心 · 2026-08-23
- OpenArm mjlab: Open-source robot manipulation env — neurosp1ke · 2026-08-23
- Nature paper: Brain-guided LLMs improve robust reasoning — Dr_Alex_Crimi · 2026-08-23