DeepMind paper fixes Softmax and cross-entropy gradient loss from fp rounding
jiqizhixin · x · 2026-11-06
Google DeepMind's Jonathan T. Barron, Dan Gnanapragasam and Michael Tschannen explain two puzzling behaviors in a preprint, "A Fix for Softmax and Cross-Entropy": why an already-correct classifier can lose its target gradient, and why identical teacher and student models keep drifting apart during distillation.
- Bottleneck: floating-point rounding can turn a confident target probability into exactly one, prematurely zeroing its gradient; subtracting nearly identical teacher/student probabilities leaves spurious residual gradients.
- Fix: reorganize the same functions in log space using softplus, expm1, and log1p to retain small quantities — meaningful gradients preserved without switching to higher precision; identical distillation logits now yield exactly zero gradient.
- Payoff: demonstrated on an fp32 modulo-31 addition experiment with AdamW (weight decay 1).
More from Research
- Anthropic report: full AI automation could yield 15-16x research output, labs remain the bottleneck — teortaxesTex · 2026-11-06
- Elastic's VP of AI: Treat search as test-time compute, latency is a currency to spend — gaganghotra_ · 2026-11-06
- Training-Free Speculative Decoding Cuts Latency for Full-Duplex Speech LMs — CUHKSZ · 2026-11-06
- TailSteer: Drift-Resistant Multi-Turn Image Editing from CUHK MMLab — MMLab-CUHK · 2026-11-06
- EvoMemState: State-Centric Long-Term Memory for Personal Agents — Zhishang Xiang · 2026-11-06
- Oxford Position Paper: From Autonomous Agents to Cognitive Symbionts — Oxford · 2026-11-06