A look back at optimizer search: Lion boosted ViT accuracy 2% and cut JFT pretraining compute 5x
akbirthko · x · 2026-10-24
The author revisits research on discovering optimizers via program search, citing Lion (Symbolic Discovery of Optimization Algorithms, arXiv:2302.06675) and VeLO.
Lion, from Google, keeps only momentum and is more memory-efficient than Adam: up to 2% ImageNet accuracy gains for ViT, up to 5x pretraining compute savings on JFT, and up to 2.3x less compute for diffusion training. The author notes that varying costs and value estimates would be an interesting follow-up problem.
More from Research
- AI System Matches Ovarian Cancer Organoids to Therapies Before Treatment — vaibhavbetter · 2026-10-24
- Do Benchmarks Really Reflect Real-World Workflows? Four Objections — Lucky_Creme_5208 · 2026-10-24
- Primitive Research's AI pipeline finds 49 exoplanet candidates from 19 million TESS stars — altryne · 2026-10-24
- New paper introduces "bottleneck pressure" to find where AI raises the value of work it can't do — soumitrashukla9 · 2026-10-24
- Dev rebuilds old paper into Synthetic Social Graph, an interactive social-dynamics energy-field simulator — seanmcdonaldxyz · 2026-10-24
- UC Berkeley's 'Modern Computer Vision' Course Lectures Now Freely Available Online — burny_tech · 2026-10-24