New paper: KL, JSD and entropy implementations share softmax's numerical instability
brandondamos · x · 2026-11-17
Brandon Amos released a paper showing that standard divergence operations — forward/reverse KL, generalized Jensen-Shannon divergence, Monte Carlo KL estimates, and entropy — suffer from the same rounding and cancellation issues recently identified in softmax/soft cross-entropy when distributions are nearly equal, affecting both forward and gradient computations in settings like knowledge distillation, RL, and proximal methods. The paper proposes numerically stable reformulations with example implementations validated by property-based tests.
More from Research
- Google DeepMind unveils Gemini 4 Carbon for physics, math and life science research — GoogleDeepMind · 2026-11-17
- Gemini 4 Carbon proves Signal crypto security in weeks, finds 40x more protein sequences — GoogleDeepMind · 2026-11-17
- Frame-by-frame real gradient descent: α=0.6 zigzags the valley, α=0.3 glides down — ZeroStateReflex · 2026-11-17
- Bayesian Statistics Explained: Probability as Belief, Not Just Frequency — mdancho84 · 2026-11-17
- Data Scientist Distills 2 Years of Bayesian Statistics Lessons Into a 2-Minute Primer — mdancho84 · 2026-11-17
- 'Formalizing All Math' went from absurd to consensus in 8 months; CFSG audit found 96 paper errors — AlexKontorovich · 2026-11-17