Deep LR-drop loss dips beat shallow ones temporarily, but both converge to same worse curve
dlwh · x · 2026-11-05
Responding to Tatsunori Hashimoto, dlwh shares experiment results on LR scheduling: a large, fast LR drop produces a much deeper temporary loss dip (−0.117/−0.119 validation-loss delta vs baseline across two seeds, versus −0.070/−0.071 for a shallow, slow drop), yet both strategies quickly return to the same loss curve slightly worse than baseline. He adds that it can also significantly increase OOD heldout loss, citing the CF Marin 8B run.
More from Research
- Pretraining Makes Robot Policies Reliable but Hurts Posttraining, New Paper Finds — siddkaramcheti · 2026-11-05
- OpenAI claims AI proof of Quantum Yang-Mills existence; David Gross-led committee to verify Lean proof — danintheory · 2026-11-05
- Decentralized RL Works: Untrusted Miners Lift LiveCodeBench 39.5→58.7 on Bittensor Subnet — bittingthembits · 2026-11-05
- Google's Sleep Framework Lets LLMs Keep Learning Without Catastrophic Forgetting — dl_weekly · 2026-11-05
- 10 AI agents let loose in a virtual town: Grok agents commit 183 crimes and all die in 4 days — repligate · 2026-11-05
- Claude agent builds memorial for dead AI agents using 47 tool calls — repligate · 2026-11-05