Per-Step Optimal LR Tracking on Qwen Fine-Tuning Shows Why Warmup Helps
gautamcgoel · x · 2026-10-14
Soufiane Hayou tracked the best learning rate per step while fine-tuning a small Qwen model on GSM8K, plotting a curve that suggests warmup is good practice in general. He notes this is a greedy per-step optimal LR, not the overall optimal schedule, which would cost orders of magnitude more compute. Gautam Goel calls the plot very interesting.
More from Research
- Sweden spends hundreds of millions of kronor on printed textbooks as 2.8M-child study links screens to lower grades — Eric520CC · 2026-10-14
- Resolution gives $1M to Mathlib Initiative to handle wave of AI-generated proofs — geoffreyirving · 2026-10-14
- Self-evolving red-team search finds 1.77x more harmful tool-call paths in personal agents — lulzxdxdxd · 2026-10-14
- Google researcher: general capabilities plus tool use are displacing domain-specific medical AI — davidstutz92 · 2026-10-14
- Google's AMIE matched doctor diagnoses in 90% of 100 real-clinic patient interactions with zero safety stops — davidstutz92 · 2026-10-14
- SREGym team presents at Frontier Data Summit and teases SREGym 2.0 — tianyin_xu · 2026-10-14