Google researchers find two numerical bugs in softmax/cross-entropy, fixing Slingshot and grokking issues
jon_barron · x · 2026-10-24
Google's Jonathan T. Barron and co-authors published an arXiv paper, "A Fix for Softmax and Cross-Entropy," identifying two numerical bugs in standard implementations of softmax, log-softmax, and cross-entropy:
- Softmax Collapse: when the target logit is much larger than the others, its gradient rounds to zero, so gradients no longer sum to zero.
- Softmax Cancellation: when student logits are close to teacher logits (e.g., distillation initialized with student = teacher), non-zero gradients appear even when the two are identical.
The paper presents simple fixed implementations whose effects include: eliminating the previously observed "Slingshot" phenomenon, speeding up grokking with weight decay, preventing accuracy collapse without weight decay, stopping student networks from drifting away from the teacher during distillation, and showing that post-memorization logit growth is bounded by an equilibrium between weight decay and Adam's ε hyperparameter.
More from Research
- Frequency drives conventionalization: higher-exposure communities conventionalize more words — abenitezburraco · 2026-10-24
- GeoWind2Plan: seconds-scale 3D urban wind prediction for energy-efficient drones — AnimaAnandkumar · 2026-10-24
- Flowing water and lava come to 3D Gaussian Splatting scenes — jonstephens85 · 2026-10-24
- Immorta Bio's AI-designed 9-amino-acid peptide SenoMune-0257 targets senescent cells — DavidmComfort · 2026-10-24
- France rebuilt its tax code as MLang, a formally verified open-source compiler — fkasummer · 2026-10-24
- Running RFD4-Proteina protein design model at thinking_high on consumer cloud GPUs — AllThingsApx · 2026-10-24