Hashimoto's CS312 quiz: what happens when you dip the LR mid-training
tatsu_hashimoto · x · 2026-11-05
Tatsunori Hashimoto shares a CS312 (Deep Learning Alchemy) prediction question on training dynamics: train a 35M-param LM with a WSD schedule (peak LR 0.003), then temporarily lower the LR mid-training and restore it. How does validation loss change vs the baseline?
Two matched schedules are compared — a short deep dip (40-50% of training, down to 0.1x) vs a longer shallow dip (35-55%, down to 0.55x), with the same total removed LR. The quiz probes the 'river valley' mental model of LLM training, with four qualitative predictions to choose from.
Related event: Tarumoto's LR Quiz: Mid-Training Drop Yields Only Temporary Gains(2 posts)→
More from Research
- Michael Levin preprint proposes topoiesis mechanism for group morphogenesis rescue — drmichaellevin · 2026-11-05
- MoE beats dense in matched OOD finetune: DeepSeek V3.1 hits 0.804 CE vs Llama 405B — kalomaze · 2026-11-05
- diffTLM: a differentiable vocal tract simulator wins NeurIPS 2026 Spotlight — vincesitzmann · 2026-11-05
- The CERN analogy fits robot evaluation: shared hardware, protocols and published failures — DJiafei · 2026-11-05
- Google Research unveils Agile Deliberation, an LLM framework for teaching VLMs subjective categories — RanjayKrishna · 2026-11-05
- Firecrawl launches Legal Retrieval Benchmark: its index tops 581-request test at 76.2% — devdigest · 2026-11-05