Experiment Shows Mid-Training LR Drop Yields Only Temporary Loss Improvement

Tatsunori Hiramoto's experiment on a 35M-parameter model with WSD scheduling showed that deeply dropping the learning rate mid-training yields a larger temporary loss improvement, but both strategies converge to the same slightly worse-than-baseline curve after the LR recovers. Similar results were independently confirmed by dlwh.

2026-11-05 ~ 2026-11-05 · 3 related posts

• Hashimoto's CS312 quiz: what happens when you dip the LR mid-training — tatsu_hashimoto · 2026-11-05
• Hashimoto's answer: deep LR dips pay big briefly, then converge — tatsu_hashimoto · 2026-11-05
• Deep LR-drop loss dips beat shallow ones temporarily, but both converge to same worse curve — dlwh · 2026-11-05