New paper: KL, JSD and entropy implementations share softmax's numerical instability

brandondamos · x · 2026-11-17

Brandon Amos released a paper showing that standard divergence operations — forward/reverse KL, generalized Jensen-Shannon divergence, Monte Carlo KL estimates, and entropy — suffer from the same rounding and cancellation issues recently identified in softmax/soft cross-entropy when distributions are nearly equal, affecting both forward and gradient computations in settings like knowledge distillation, RL, and proximal methods. The paper proposes numerically stable reformulations with example implementations validated by property-based tests.

Original post →

More from Research

Research channel →