Stanford CS312 quiz: can LR or weight decay rescaling compensate for an 8× larger batch?
tatsu_hashimoto · x · 2026-10-28
Hashimoto posts a prediction quiz from Stanford's new CS312 "Deep Learning Alchemy" course: with a 35M-param Transformer trained on 614.4M tokens (AdamW, reference best at batch 64: LR 0.003, WD 0.2, loss 2.9178), which recipe gives the lowest val loss after raising batch size to 512 — keep both, LR×√8, LR×8, WD×8, or both×√8? Part of the course's optimizer hyperparameter unit testing scaling intuition.
Related event: Stanford CS312 Quiz: What to Tune When Batch Size Grows 8x(2 posts)→
More from Research
- Eric Topol highlights immunome paper combining multi-assay data with AI to model immune aging — EricTopol · 2026-10-28
- Google Research's 'sleep framework' lets LLMs keep learning, retaining up to 10M tokens — behrouz_ali · 2026-10-28
- CATCH: ~200 examples of adversarial training make LLMs robustly honest across benchmarks — irinarish · 2026-10-28
- Andrew Critch's dubious credences: P≠NP at 80%, Riemann hypothesis at just 60% — AndrewCritchPhD · 2026-10-28
- AI-driven Indian biotech PopVax, backed by $20M from Vitalik's Balvi fund, enters human trials with fridge-stable broad-spectrum Covid vaccine — chaitjo · 2026-10-28
- Cognition launches FrontierCode 2.0, a coding benchmark testing judgment beyond the spec — DevinAI · 2026-10-28