Stanford CS336 notes: Percy Liang on scaling recipes and hyperparameter transfer
stanfordnlp · x · 2026-10-31
Notes from Stanford's CS336 (Language Modeling from Scratch, Spring 2026, taught by Percy Liang and Tatsu Hashimoto) capture the key mindset shift in Liang's scaling laws lecture: stop thinking about one model and think of a scaling recipe — a rule mapping compute budget to hyperparameters. Run small-scale experiments (up to 1e24 FLOPs), fit a loss curve, and extrapolate to predict the big run before spending the compute. The catch: this only works if small-scale optima transfer — the best learning rate shifts with model width, which is why hyperparameter transfer (scaling initialization and per-layer LRs with width, Yang+ 2022) matters. The course walks through the full pipeline of building an LM from scratch, with lectures on YouTube.
More from Research
- NVIDIA Nemotron 3.5 scales LLM self-verification, +9-10 points on agentic benchmarks — sarahcat21 · 2026-10-31
- Unofficial OpenTargets MCP Server Brings Gene-Drug-Disease Data to LLMs — modelcontextprotocol · 2026-10-31
- Researchers find 5 personality structures in LLM weight space, Big Five-like — luke_drago_ · 2026-10-31
- NVS benchmarks are misaligned: proper pose alignment puts 3D Gaussians back at SOTA — kwangmoo_yi · 2026-10-31
- arXiv moderator: submission surge causes months-long holds, new moderators wanted — tdietterich · 2026-10-31
- arXiv policy checks reject class projects, proposals, surveys and position papers — tdietterich · 2026-10-31