Stanford experiments on weight averaging and EMA: why it cuts loss but loses to LR decay
stanfordnlp · x · 2026-11-21
Stanford NLP reshared @ginojirat's series of experiments in CS312 on weight averaging and EMA in pretraining:
- Theory (the river-valley landscape): training progresses along a "river" direction (real progress) while oscillating sharply in "valley" directions (cosine similarity of consecutive updates along the top Hessian eigenvector is -1). Averaging (EMA or otherwise) cancels the fast-varying valley directions and keeps the river direction — which is why averaging recent checkpoints lowers loss.
- Finding: on their model organism, EMA slightly loses to LR decay. EMA maintains a lagging copy of progress along the river direction, while LR annealing descends the valley without averaging in stale checkpoints.
- Caveat: this holds at the edge of stability under full-batch gradient descent; the SGD case is still being explored.
The thread also cites @datologyai's practice: skipping LR decay in favor of a running weight average, merging pre/post-context-extension models for ablation gains, and using merges as a stand-in for MOPD in their RLVR recipe.
More from Research
- Stanford CS329Z Lecture 9 Released: Optimizing AI Agents via Prompts vs. Weights — stanfordnlp · 2026-11-21
- Hardy's postcard: claiming he'd proved the Riemann hypothesis to keep the boat from sinking — arampell · 2026-11-21
- Mathematician predicts superhuman AI math in 2-3 years after LLM disproves unit-distance conjecture — soumitrashukla9 · 2026-11-21
- Universal Dependencies v2.19 Released: 49 New Treebanks, 33 New Languages — stanfordnlp · 2026-11-21
- CoRL 2026 best paper SuperMimic accused of repackaging 2019-era work with slick marketing — animesh_garg · 2026-11-21
- ELLIS Institute Tübingen Brings 48 Papers, 2 Orals to Three-City NeurIPS 2026 — maksym_andr · 2026-11-21