BF16 rounding breaks a conservation law, blowing up FlashAttention gradients late in training
HongyiWang10 · x · 2026-10-04
A hidden gradient bug in BF16 attention
A Rutgers team pretrained a 450M-parameter transformer on 50B tokens with FlashAttention-3 in BF16. Training was healthy for 25B tokens, then the gradient norm grew 1000× and the final loss ended 0.2 nats above FP32 attention — without a single NaN. Healthy-looking training does not mean correct gradients.
- Cause 1: a fused multiply-add rounding issue in the forward softmax (previously treated as an extreme-input NaN case, never fixed in FlashAttention-3).
- Cause 2: BF16 rounding breaks the conservation law that softmax score gradients sum to zero along each row, leaking the mean key into the query gradient — a leak that grows exactly as late training makes keys large and attention sharp.
Fix: GProj
The authors propose GProj (gauge projection), restoring the zero-sum law after rounding. Query/key gradient accuracy and final loss match FP32 attention with only 4.7% extra step time. Recomputing just two layers' attention backward in FP32 removes nearly all excess gradient.
> Training looks fine? Check the backward pass anyway.
More from Infra
- Ex-Waymo exec Abe Murray: CUDA and ISAs won't stay moats, huge chip innovation incoming — IgorCarron · 2026-11-20
- MIT and NVIDIA's QDA uses agents to auto-design LLM quantization recipes, beating top public ones by up to 2.2 points — omarsar0 · 2026-11-20
- Caixin: US remote-access cutoff and Nvidia CEO exit leave China's tech giants halting overseas compute deals — Gregory_C_Allen · 2026-11-20
- QuixiInfer: Rust Engine Runs 125B MoE Model on a Single RTX 5090 — QuixiAI · 2026-11-20
- QuixiInfer: from-scratch Rust engine runs DeepSeek with 256k context on a single RTX 5090 at ~30 tok/s — QuixiAI · 2026-11-20
- AMD's Lisa Su: data center to hit ~80% of revenue by 2027, sees $220B server CPU market — AccBalanced · 2026-11-20