All existing softmax/cross-entropy implementations have a subtle bug, says Google's Jon Barron
jon_barron · x · 2026-10-24
Jon Barron shares a new result: all existing implementations of softmax and cross-entropy — JAX, PyTorch, textbooks — contain two subtle bugs. A simple rewrite fixes both and eliminates "slingshot" spikes during grokking.
The first bug, "Softmax Collapse" (identified by Prieto et al. 2025), occurs when one logit dominates: its gradient silently rounds to zero while others don't, so gradients no longer sum to zero. The post includes a full JAX rewrite of logsoftmax and crossentropy that keeps gradients well-behaved across float64/32/bfloat16 precisions.
More from Research
- Stanford-led paper traces 'bigger is better' to data-induced competition for neurons — niloofar_mire · 2026-10-24
- Frequency drives conventionalization: higher-exposure communities conventionalize more words — abenitezburraco · 2026-10-24
- GeoWind2Plan: seconds-scale 3D urban wind prediction for energy-efficient drones — AnimaAnandkumar · 2026-10-24
- Flowing water and lava come to 3D Gaussian Splatting scenes — jonstephens85 · 2026-10-24
- Immorta Bio's AI-designed 9-amino-acid peptide SenoMune-0257 targets senescent cells — DavidmComfort · 2026-10-24
- France rebuilt its tax code as MLang, a formally verified open-source compiler — fkasummer · 2026-10-24