ICML paper: reaching ultra-flat regions too early can derail gradient descent training
MarkSchmidtUBC · x · 2026-10-28
A blog explainer of the ICML 2026 paper "Flatland: The Adventures of Gradient Descent with Large Step Sizes" (Galli, Bartolomaeus, Rauhut, Fox, Schmidt) challenges the folklore of setting the largest non-diverging learning rate: with variable step sizes, entering extremely flat (low-sharpness) regions too early can prevent networks from learning successfully. The paper maps how large steps can grow while training still converges, with implications for LR schedules and loss-landscape geometry.
More from Research
- TTME beats test-time training on 5 math tasks while cutting FLOPs by 82.8-84.6% — UCSantaBarbara · 2026-10-28
- EnvLunch turns executable environments into free verifiable agent training data — KingsCollegeLondon · 2026-10-28
- Anthropic reveals it is developing its own experimental drugs using Claude — Polymarket · 2026-10-28
- Stanford launches TaskNet: 132 worker interviews build a large-scale occupational task inventory — soumitrashukla9 · 2026-10-28
- KAIST's Principia uses LLMs to construct and refine scientific principles at test time — coallaoh · 2026-10-28
- The AlphaFold Data Problem: Quantum Chemistry Models as Training Data for Larger Scales — TimothyDuignan · 2026-10-28