DeepMind paper exposes two numerical bugs in standard softmax and cross-entropy
kalyan_kpl · x · 2026-10-25
A new paper by Jonathan T. Barron and colleagues at Google DeepMind identifies two numerical bugs in standard softmax, log-softmax, and cross-entropy implementations:
- Softmax Collapse: when the target logit is much larger than the others, its gradient rounds to zero, so gradients no longer sum to zero.
- Softmax Cancellation: near-equal student and teacher logits still produce non-zero gradients.
The authors present simple fixed implementations that eliminate the previously observed "Slingshot" phenomenon, speed up grokking under weight decay, prevent accuracy collapse without weight decay, stop student networks drifting from teachers during distillation, and show that post-memorization logit growth is bounded by an equilibrium between weight decay and Adam's ε hyperparameter.
More from Research
- New IEEE IT paper: geometric Jensen-Shannon divergence between Gaussian measures on Hilbert space — FrnkNlsn · 2026-10-28
- CPR-Dataset: retrievers fail at cross-domain discovery, training gives +70% on BRIGHT — coallaoh · 2026-10-28
- Hugging Face robot duck walks in simulated snow with 290,000 physics-simulated flakes — lukas_m_ziegler · 2026-10-28
- Mercor, Stanford and NBER launch TaskNet, an open-source task catalog updating O*NET for the AI era — soumitrashukla9 · 2026-10-28
- François Fleuret Proposes Tagging Every Token 'Heard/Thought/Said' to Solve Alignment — francoisfleuret · 2026-10-28
- Reddit builds benchmark release-date timeline showing evals go stale after 3-4 months — Express_Quail_1493 · 2026-10-28