Stanford CS312 quiz: can rescaling LR or weight decay rescue an 8x batch size increase?
2C_ornot2C · x · 2026-10-30
Tatsu Hashimoto is posting intuition quizzes from Stanford's CS312 (Deep Learning Alchemy) exams. The first covers optimizer hyperparameters: training an 8-layer 35M-parameter Transformer on 614.4M tokens with AdamW (peak LR 0.003, WD 0.2, val loss 2.9178 at batch 64), what happens when batch size grows to 512 — 8x fewer optimizer steps — and can rescaling LR or weight decay compensate? Shared as the best resource for learning how frontier LLMs are trained without working at a lab.
More from Research
- Perplexity's pplx-embed-v2-context ditches gold-chunk labels, sets SOTA on retrieval benchmarks — _reachsumit · 2026-10-30
- Gavin Crooks: Maybe it's time to write some 'last papers' as AI absorbs knowledge — MaxUnfried · 2026-10-30
- DoorDash's PALATE: GPT-style user embeddings lift ad revenue ~1% in production — _reachsumit · 2026-10-30
- A sequential decision framework unifies evaluation of LLM-guided search policies — _reachsumit · 2026-10-30
- Alibaba's PFA progressively focuses attention to speed long-sequence generative recommendation — _reachsumit · 2026-10-30
- RecRSI: ranking-scored short trials cut wasted retraining in LLM-driven recommender self-improvement — _reachsumit · 2026-10-30