Google researchers fix two numerical bugs in softmax and cross-entropy, eliminating 'Slingshot'
ZeYanjie · x · 2026-10-24
Jonathan Barron and colleagues (Google) publish an arXiv paper identifying two numerical bugs in standard softmax/log-softmax/cross-entropy implementations:
- Softmax Collapse: when one target logit is much larger than the rest, its gradient rounds to zero and gradients no longer sum to zero;
- Softmax Cancellation: near-equal student/teacher logits still produce non-zero gradients even when exactly equal.
Their simple fixed implementations eliminate the previously-observed "Slingshot" phenomenon, speed up grokking with weight decay, prevent accuracy collapse without it, stop students drifting from teachers in distillation, and show post-memorization logit growth is bounded by an equilibrium between weight decay and Adam's epsilon.
More from Research
- Frequency drives conventionalization: higher-exposure communities conventionalize more words — abenitezburraco · 2026-10-24
- GeoWind2Plan: seconds-scale 3D urban wind prediction for energy-efficient drones — AnimaAnandkumar · 2026-10-24
- Flowing water and lava come to 3D Gaussian Splatting scenes — jonstephens85 · 2026-10-24
- Immorta Bio's AI-designed 9-amino-acid peptide SenoMune-0257 targets senescent cells — DavidmComfort · 2026-10-24
- France rebuilt its tax code as MLang, a formally verified open-source compiler — fkasummer · 2026-10-24
- Running RFD4-Proteina protein design model at thinking_high on consumer cloud GPUs — AllThingsApx · 2026-10-24