Stanford CS312 quiz: scale weight decay 8x when batch size grows 8x
tatsu_hashimoto · x · 2026-10-28
Stanford CS312 (Deep Learning Alchemy) instructor Tatsu Hashimoto posted an exam question: at a fixed token budget, can rescaling LR or weight decay compensate for an 8x batch size increase for an LM?
The answer is D: keep learning rate fixed, multiply weight decay by 8x. With 8x larger batches there are 8x fewer optimizer steps, and under decoupled AdamW cumulative decay depends roughly on WD × the sum of learning rates, so LR×WD should scale 8x to preserve total decay.
Why scale WD rather than LR: in course assignments on small models, optimal weight decay extrapolated well to larger batches while optimal LR scaling did not, making WD scaling more reliable in extreme batch regimes. He notes this is specific to the tiny class model, not a universal rule.
Related event: Stanford CS312 Quiz: What to Tune When Batch Size Grows 8x(2 posts)→
More from Models
- OpenBMB open-sources MiniCPM OCR family: 1.3B model tops open-source OCR benchmarks, 2B evaluator hits 0.936 AUROC — AdinaYakup · 2026-10-28
- Users report Grok Bot latency dropped sharply, with noticeably snappier responses — mark_k · 2026-10-28
- Cognition's FrontierCode 2: Opus 5.5 and GPT-6 Astra lead; Chinese OSS models lag at 17% — scaling01 · 2026-10-28
- Statistically Lossless Qwen3.8 27B GGUF Quants Hit ~99% EAR at 23GB, Feels FP8-like — Dany0 · 2026-10-28
- mark_k pans rumored OpenAI Terra and Astra models, calls for a stronger Sol tier — mark_k · 2026-10-28
- Kimi K3.1 teased to adopt DSA like DeepSeek V3.2 for better inference economics — zephyr_z9 · 2026-10-28