Softmax/CE bug found in all implementations; KL and JSD share same numerical flaws
LotfiSanae · x · 2026-11-17
- Jon Barron reports that all existing implementations of softmax and soft cross-entropy (JAX, PyTorch, textbooks) contain two subtle bugs; a simple rewrite fixes both and eliminates "slingshots" during grokking.
- Following up, Brandon Amos (Reflection AI) posted a paper, On numerical issues of divergences between nearly equal distributions (arXiv:2611.18511), showing similar issues in forward/reverse KL divergence, generalized Jensen-Shannon divergence, Monte Carlo KL estimates, and entropy.
- When distributions are nearly equal, rounding and cancellation errors corrupt both forward computations and gradients — potentially silently affecting knowledge distillation, reinforcement learning, and proximal methods for years.
- The paper proposes numerically stable reformulations with example implementations, validated by property-based tests.
More from Research
- Google DeepMind unveils Gemini 4 Carbon for physics, math and life science research — GoogleDeepMind · 2026-11-17
- Gemini 4 Carbon proves Signal crypto security in weeks, finds 40x more protein sequences — GoogleDeepMind · 2026-11-17
- Frame-by-frame real gradient descent: α=0.6 zigzags the valley, α=0.3 glides down — ZeroStateReflex · 2026-11-17
- Bayesian Statistics Explained: Probability as Belief, Not Just Frequency — mdancho84 · 2026-11-17
- Data Scientist Distills 2 Years of Bayesian Statistics Lessons Into a 2-Minute Primer — mdancho84 · 2026-11-17
- 'Formalizing All Math' went from absurd to consensus in 8 months; CFSG audit found 96 paper errors — AlexKontorovich · 2026-11-17