Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection
Ronald Richman, Mario V. Wüthrich
cs.LG
2026-08-19
ROR holds short optimizer tournaments on one evolving run. One-epoch scouting stays near the best fixed optimizer on four tasks at 24–35% of exhaustive-search cost.
An optimizer is usually locked in before training and kept for the whole run. Treating that choice as a hyperparameter means finishing a full run for Adam, Muon, Lion, and the rest, then throwing away every loser. The optimizer that wins the complete run need not be best at every stage: early descent and late refinement can want different update rules.
Repeated Optimizer Resampling (ROR) searches during one evolving trajectory. The claim under test is narrower than a changing schedule beating every fixed optimizer. It asks whether short tournaments can approach the hindsight-best fixed run without paying for all nine complete trainings.
A tournament has two clocks. The scouting period s is how many epochs each candidate trains from the current weights. The retained segment b is how far the winner actually advances the trajectory, with 1 ≤ s ≤ b. At the start of a round the current weights are copied K times. All nine optimizers scout for s epochs on the same data order. The best validation scout continues for b−s epochs. If the completed segment improves the incumbent validation objective by more than δ, it becomes the new incumbent. Otherwise the round is rejected and training stops.
Round cost is b+(K−1)s epoch-equivalents, counting discarded scouts. With K=9 and b=3, s=1 costs 11 and s=3 costs 27. Exhaustive search costs the sum of the nine fixed runs to their own stopping points.
Two state policies:
Losing scouts do not keep their state. That state was learned on discarded weights; attaching it to the winner would be a transplant. Keeping nine matching trajectories would recreate the exhaustive search ROR is meant to avoid.
A one-shot baseline scouts once for one epoch, then continues the winner from a freshly initialized optimizer until ordinary early stopping. The optimizer set is never compared again.
The four tasks are MNIST, Fashion-MNIST, and two French motor-insurance claim-count models (a 4,791-parameter MLP and a 4,433-parameter CLS Transformer). The image model is a 300,938-parameter fully connected net, not a CNN. Learning rates are fixed in advance: 10⁻³ for Adam, AdamW, Nadam, and RMSprop; 3×10⁻⁴ for Lion; 5×10⁻² for SGD-Nesterov; 2.5×10⁻³ for ScheduleFreeAdamW. Muon updates only hidden two-dimensional kernels; biases and input/output projections go through its AdamW branch. Ten shared seeds, validation-based selection, test set unread until the end.
Shorter scouting is the cost result that holds. One-epoch ROR uses 24% to 35% of the training needed to finish all nine fixed runs, and 51% to 68% less than s=3. Across all four tasks, every paired 95% interval between ROR and the best fixed optimizer includes zero. The same is true for SP-ROR versus CS-ROR.
| Task | Best fixed | One-epoch ROR | Exhaustive cost | s=1 cost |
| MNIST | Muon 98.095% | CS 97.948% / SP 97.988% | 150.7 | 37.4 / 39.6 |
| Fashion-MNIST | SF-AdamW 89.059% | CS 89.007% / SP 88.992% | 152.4 | 53.9 / 48.4 |
| Insurance MLP | Lion 0.238365 | CS 0.238500 / SP 0.238476 | 442.4 | 107.8 / 110.0 |
| Insurance Transformer | Lion 0.237817 | CS 0.237627 / SP 0.237646 | 140.8 | 42.9 / 40.7 |
On the insurance Transformer, one-epoch ROR posts the lowest observed means, still with a paired interval versus Lion that includes zero. On MNIST, stretching the scout from 1 to 3 epochs lifts CS-ROR by about 0.184 points with an interval that excludes zero, at roughly triple the cost. Fashion-MNIST and both insurance tasks show no clear gain from the longer scout.
One-shot is cheaper than one-epoch ROR on every task. On MNIST it is also better, at 98.116% and 20.3 epoch-equivalents. On the other three tasks the repeated-tournament means are better. The schedules match that split: MNIST averages 0.5 optimizer switches, the insurance MLP 5.6. When the early ranking persists, one query captures most of the value. Repeated tournaments earn their roughly 2× cost on tasks that keep switching later.
For a new problem whose optimizer is unknown, one-epoch scouting is a cheap substitute for exhaustive search. It is not evidence that a dynamic schedule trains a better model. The Keras 3 sketch is copyable for small nets and tabular work. Whether the same tournament logic is worth the wall-clock of a large-model epoch is not tested here.
Four tasks, ten seeds, and networks no larger than a 300k-parameter MLP or a 4–5k tabular model. Learning rates were frozen before the study; a different grid could reorder both the fixed ranking and the ROR schedule. SP-ROR only continues the incumbent, so each tournament compares continuation against fresh challengers, not nine exact histories.
The same validation split is queried at every tournament, so the schedule can overfit that holdout. ROR stops at the first rejected round, while fixed baselines get five epochs of patience. That asymmetry is harsh on ROR's accuracy and flattering to its cost. Epoch-equivalents also ignore that a Muon step does more linear algebra than an Adam step.