New study traces late LM pretraining loss spikes to output-layer rounding errors
micahgoldblum · x · 2026-11-06
Work by Chen et al. traces late-emerging loss spikes in LM pretraining to rounding errors in the output layer: under high learning rates the output logit norm keeps growing until loss spikes. The paper also shows why three common fixes — z-loss, μ-centering, and momentum SGD — help through the same mechanism. Authors include Tom Goldstein, John Kirchenbauer, and Micah Goldblum.
More from Research
- 12B open Nemotron trained on Surge RL data beats GPT-5.6 on SWE-Bench Pro V2 — echen · 2026-11-06
- Goodfire unveils mechanistic shaping: using interpretability as training feedback — teortaxesTex · 2026-11-06
- TurnWise: turn-level GRPO improves when-to-answer in streaming video RL — _akhaliq · 2026-11-06
- Google publishes EmbeddingGemma 2 tech report: 740M multimodal embedding model — bdsqlsz · 2026-11-06
- Biomni Lab API enters beta for programmatic, parallel agentic science workflows — KexinHuang5 · 2026-11-06
- AI-era organizational economics paper lands AEA 2027 slot, cited by Jason Furman — soumitrashukla9 · 2026-11-06