Why AdamW breaks training: it destroys the LM head's zero-sum gradient structure
micahgoldblum · x · 2026-11-03
Part 6 of Goldblum's thread poses the puzzle: adding a constant to every logit changes neither softmax nor cross-entropy gradients in exact arithmetic, so why does mean-logit drift break training? The cited prior post explains the cause: LM-head cross-entropy gradients sum to zero across the vocabulary, but Adam's coordinate-wise preconditioning breaks this, creating drift in the mean output embedding and mean logit.
Related event: AdamW-Induced Logit Drift Fails under BF16 Finite Precision(2 posts)→
More from Research
- New paper: exponential residual flows bring probability-flow ODEs to discrete diffusion — ArnaudDoucet1 · 2026-11-04
- Researchers Push Back: Frontier Labs Are Overlooking Chemistry in AI for Science — anshulkundaje · 2026-11-04
- Coding agents build baselines for 36 top-journal AI methods, matching ~95% of scores — owl_posting · 2026-11-04
- EGA beats PCA at recovering dimensions from LLM item embeddings, study finds — GolinoHudson · 2026-11-04
- Speculative decoding math explained: acceptance rate equals 1 minus TV distance — techNmak · 2026-11-04
- AgentTime Adds Combined Time-Awareness Score; GPT Models Clearly Ahead — maksym_andr · 2026-11-04