AdamW-Induced Logit Drift Fails under BF16 Finite Precision

Goldblum explained why average logit drift breaks LM training despite being mathematically inert in exact arithmetic: in BF16, larger magnitudes lose the resolution to represent differences between logits, which determine the softmax distribution.

2026-11-03 ~ 2026-11-03 · 2 related posts

• Why AdamW breaks training: it destroys the LM head's zero-sum gradient structure — micahgoldblum · 2026-11-03
• BF16 loses resolution as mean logit grows, breaking softmax probabilities — micahgoldblum · 2026-11-03