Stanford experiments on weight averaging and EMA: why it cuts loss but loses to LR decay

stanfordnlp · x · 2026-11-21

Stanford NLP reshared @ginojirat's series of experiments in CS312 on weight averaging and EMA in pretraining:

The thread also cites @datologyai's practice: skipping LR decay in favor of a running weight average, merging pre/post-context-extension models for ablation gains, and using merges as a stand-in for MOPD in their RLVR recipe.

Original post →

More from Research

Research channel →