Softmax's standard implementations hide two numerical bugs, Google paper shows
srush_nlp · x · 2026-10-28
A paper by Jonathan T. Barron et al. identifies two numerical bugs in standard softmax/log-softmax/cross-entropy implementations: Softmax Collapse (gradient of a dominant logit rounds to zero, so gradients no longer sum to zero) and Softmax Cancellation (non-zero gradients even when student and teacher logits are equal). The authors present simple fixed implementations that eliminate the "slingshot" phenomenon, speed up grokking with weight decay, prevent accuracy collapse without it, and stop student drift during distillation. The surrounding discussion notes TorchLean had the same bug, using it as a lesson in the mindset needed for formal verification.
Related event: Google Paper Reveals Two Numerical Bugs in All Softmax Implementations(3 posts)→
More from Research
- RobotAgentBench debuts to test open-world robot autonomy: manipulation ≠ autonomy — HennyJieCC · 2026-10-28
- Would you ship a policy trained mostly in generated worlds? AIFlow polls robot people — AIFlow_ML · 2026-10-28
- Ranking Attention Heads by Gradient Norm Enables Sparse RL Fine-Tuning — SimonShaoleiDu · 2026-10-28
- Researchers predict arXiv will become a distribution platform as preprints move to GitHub — mjuric · 2026-10-28
- DeepMind caveat: robot grasp policies drift on object permanence, sims still win — AIFlow_ML · 2026-10-28
- Carbon-X portal: burn tokens for community genomics training credits and bounties — LoubnaBenAllal1 · 2026-10-28