Simple fix for training collapse: momentum SGD on the LM head only
micahgoldblum · x · 2026-11-03
Part 10 of Goldblum's thread: the fix is to swap AdamW for momentum SGD only on the LM head, keeping AdamW elsewhere. SGD-M preserves the gradient's zero-sum structure, avoiding Adam-induced drift in the mean output embedding. Context: FP32 logits still grow but loss doesn't diverge; in BF16, quantization error grows by orders of magnitude before failure.
More from Research
- PROCS paper: parallel snapshot compression cuts inference cold-start latency up to 44.7% — charles_irl · 2026-11-04
- Hugging Face Open ASR Leaderboard drops LibriSpeech from default average WER ranking — rdesh26 · 2026-11-04
- Yoav Goldberg tempers Astra weight-reading hype: it used a simplified 10x10 Breakout — yoavgo · 2026-11-04
- DeepMind RL-trained Gemini doubles correct psychiatric differential diagnoses in blind study — SRSchmidgall · 2026-11-04
- HF researcher: auditing online eval biases turned into the biggest section of my BP — antoine_chaffin · 2026-11-03
- Researchers: many big-lab model evals only prove a data mixture gap, not capability limits — josephdviviano · 2026-11-03