Reactive optimizers: an agent rewrites optimizer code mid-training, beating best-practice recipes
tensorqt · x · 2026-10-22
Quoting Vincent Sitzmann's team's reactive optimizers: an agent reactively adapts the optimizer code during a training run, acting like a team of scientists that picks the best algorithm, learning rate, and momentum based on its own analysis of step-to-step training dynamics.
The poster adds they used a similar approach (mainly for LR) while training violetto, in both pre-training and post-training. Per the original thread, the method outperforms standard best-practice recipes across all tested workloads and the agent gains optimizer "superpowers" no human could have designed.
More from Models
- Polymarket odds: Anthropic 63% to have year-end #1 AI model, Meta just 9% — Polymarket · 2026-10-22
- Users say proactive Claude feels far better than the docile wait-for-input default — altryne · 2026-10-22
- Audio and video are AGI's blind spot: models can't verify what they hear or see — coldrolledpotmetal · 2026-10-22
- Huge context windows, not smarts, may be why AI cracked math, researcher argues — DimitrisPapail · 2026-10-22
- Writer Stuffs Paragraph With AI Style Markers — Pangram Still Scores It 100% Human — BecauseCulture · 2026-10-22
- Palantir touts sovereign AI: fine-tuned Nemotron beats frontier models at ~1/20 cost — BrettKrieger12 · 2026-10-22