Google researcher says all softmax and cross-entropy implementations have two numerical bugs

Google researcher Jon Barron and colleagues posted an arXiv paper on October 24, "A Fix for Softmax and Cross-Entropy," claiming that every existing implementation of softmax and cross-entropy—including JAX, PyTorch, and the textbook-standard formulation—contains two subtle numerical bugs that a simple rewrite can fix at once.

Confirmed

• The first bug relates to the "slingshot" phenomenon during grokking: Barron showed training curve comparisons where the standard cross-entropy's training loss oscillates repeatedly during the grokking phase while the fixed version converges smoothly; the fix not only removes the slingshot spikes but also speeds up grokking.
• The second bug is dubbed "Softmax Cancellation": when the model's output logits are very close to the soft-target logits (e.g., in distillation where the student is initialized as the teacher), existing implementations produce a non-zero gradient even when student and teacher are identical.
• Barron added that Adam's default eps=1e-8 happens to land exactly on a threshold related to the bug, a coincidence worth noting.

Why it matters

• Softmax and cross-entropy are among the most fundamental and widely used operators in deep learning; if numerical flaws are pervasive at the implementation level, the impact covers virtually all training and distillation pipelines.
• The fix could change how the field interprets phenomena like grokking and slingshot—these "mysterious" dynamics may partly stem from numerical error rather than the algorithms themselves, directly affecting the credibility of related research conclusions.

2026-10-24 ~ 2026-10-24 · 6 related posts

• [source] Jon Barron: every softmax/cross-entropy implementation has two subtle numerical bugs — jon_barron · 2026-10-24
• [source] Fixing cross-entropy removes grokking spikes; Adam's default eps=1e-8 sits right past the bug — jon_barron · 2026-10-24
• [source] Softmax bug #2: distillation gradients are non-zero even when student equals teacher — jon_barron · 2026-10-24
• Google researchers find two numerical bugs in softmax/cross-entropy, fixing Slingshot and grokking issues — jon_barron · 2026-10-24

2 near-duplicate retellings: jon_barron · ZeYanjie