Hashimoto's answer: deep LR dips pay big briefly, then converge
tatsu_hashimoto · x · 2026-11-05
Answering his own CS312 quiz, Tatsunori Hashimoto reveals option B wins: the bigger LR drop gives a large temporary win, but both schedules quickly return to the same loss curve — slightly worse than baseline.
Data across two seeds: the deep dip (0.1x) reaches -0.117/-0.119 validation loss vs baseline during the dip, versus -0.070/-0.071 for the shallow dip; yet post-rewarm overshoots are remarkably similar: +0.0103/+0.0092 vs +0.0115/+0.0120.
The explanation is 'river valley' dynamics common in LLM training: lower LR lets the model descend the valley walls for a bigger temporary improvement, but once LR is raised back, the trajectory oscillates back up the sides — the gains wash out. The river-valley mental model predicts these outcomes.
More from Research
- Pretraining Makes Robot Policies Reliable but Hurts Posttraining, New Paper Finds — siddkaramcheti · 2026-11-05
- OpenAI claims AI proof of Quantum Yang-Mills existence; David Gross-led committee to verify Lean proof — danintheory · 2026-11-05
- Decentralized RL Works: Untrusted Miners Lift LiveCodeBench 39.5→58.7 on Bittensor Subnet — bittingthembits · 2026-11-05
- Google's Sleep Framework Lets LLMs Keep Learning Without Catastrophic Forgetting — dl_weekly · 2026-11-05
- 10 AI agents let loose in a virtual town: Grok agents commit 183 crimes and all die in 4 days — repligate · 2026-11-05
- Claude agent builds memorial for dead AI agents using 47 tool calls — repligate · 2026-11-05