Google DeepMind finds two numerical errors in standard softmax and cross-entropy
burkov · x · 2026-10-25
A Google DeepMind article identifies two numerical errors in standard softmax and cross-entropy calculations that can distort training even when the formulas are mathematically correct:
- One removes the gradient for a highly confident correct prediction while leaving small gradients for other classes.
- The other produces inaccurate, sometimes nonzero gradients when a student model's predictions nearly match a teacher's (distillation).
These matter because adaptive optimizers like Adam can magnify tiny gradient errors into substantial training changes. The authors derive stable replacements for softmax, log-softmax, and cross-entropy that preserve accurate gradients in both low-precision and standard floating-point formats, then validate them experimentally.
More from Research
- Paper: "bottleneck pressure" metric shows where AI-infeasible work gains value as AI speeds up the rest — soumitrashukla9 · 2026-10-28
- New IEEE IT paper: geometric Jensen-Shannon divergence between Gaussian measures on Hilbert space — FrnkNlsn · 2026-10-28
- One-page Q-learning notes with math, code and a playground for RL beginners — East-Muffin-6472 · 2026-10-28
- CPR-Dataset: retrievers fail at cross-domain discovery, training gives +70% on BRIGHT — coallaoh · 2026-10-28
- Hugging Face robot duck walks in simulated snow with 290,000 physics-simulated flakes — lukas_m_ziegler · 2026-10-28
- Mercor, Stanford and NBER launch TaskNet, an open-source task catalog updating O*NET for the AI era — soumitrashukla9 · 2026-10-28