FULL STORY

Softmax and Cross-Entropy Numerical Bugs: From Reveal to Paper

Google DeepMind researchers revealed two numerical bugs in softmax and cross-entropy implementations, sparking discussion, then published an arXiv paper detailing the fixes.

2026-10-24 ~ 2026-10-28 · 2 episodes · 24 posts

Episode 1 · Google's Barron Reveals Two Numeric Bugs in All Softmax/Cross-Entropy Implementations (2026-10-24, 21 posts)

On October 24, Google/DeepMind researchers Jonathan T. Barron, Dan Gnanapragasam and Michael Tschannen released an arXiv paper, "A Fix for Softmax and Cross-Entropy," showing that every existing softmax, log-softmax and cross-entropy implementation — including JAX, PyTorch and textbook formulations — contains two subtle numeric bugs fixable by one simple rewrite. The fix was validated on grokking training dynamics, and FlashAttention's backward pass is also affected. Barron announced the results himself across multiple posts, with the news spread by reposts and retellings from ZeYanjie, codestar, IgorCarron, burkov, lmoroney and others.

Confirmed

  • The first bug, termed "Softmax Collapse" per lmoroney's retelling, triggers when one logit is far larger than the rest; burkov notes it zeroes out gradients for high-confidence correct predictions while other classes retain small gradients. Barron's training curves show standard cross-entropy loss oscillating during grokking while the fixed version converges smoothly, eliminating slingshot spikes and accelerating grokking.
  • The second bug, "Softmax Cancellation": when model logits are very close to soft-target logits (e.g., a distilled student initialized from its teacher), existing implementations produce non-zero gradients even when student equals teacher.
  • Barron notes that Adam's default eps=1e-8 happens to hit the bug-related threshold — a notable coincidence.
  • Per reposted information, FlashAttention's backward pass is also affected, broadening the blast radius beyond direct softmax/cross-entropy calls.
  • Barron also disclosed his AI-assisted workflow: the bug was first found and fixed by Gemini Flash 3.7, most experiments were run by Flash 3.8, and Gemini 4 Argon wrote the initial draft after being fed his old papers for style, before Barron rewrote it by hand.

Why it matters

  • Softmax and cross-entropy are among the most fundamental, widely used operators in deep learning; a pervasive implementation-level numeric flaw affects virtually all training and distillation pipelines, and FlashAttention's involvement extends it further.
  • The fix may reshape how the community interprets grokking and slingshot phenomena — these "mysterious" dynamics may partly stem from numeric error rather than algorithmic essence, directly affecting the credibility of related research conclusions.

1 more related posts →

Episode 2 · Google Paper Reveals Two Numerical Bugs in All Softmax Implementations (2026-10-27, 3 posts)

Google DeepMind researchers show that all existing softmax and cross-entropy implementations, including JAX, PyTorch and textbooks, contain two numerical bugs. A follow-up discussion notes the same issue also affects KL-divergence uses such as KL regularization in reinforcement learning.