Google Paper Reveals Two Numerical Bugs in All Softmax Implementations
Google DeepMind researchers show that all existing softmax and cross-entropy implementations, including JAX, PyTorch and textbooks, contain two numerical bugs. A follow-up discussion notes the same issue also affects KL-divergence uses such as KL regularization in reinforcement learning.
2026-10-27 ~ 2026-10-28 · 3 related posts
- Episode 1: Google's Barron Reveals Two Numeric Bugs in All Softmax/Cross-Entropy Implementations(2026-10-24, 21 posts)
- Episode 2: Google Paper Reveals Two Numerical Bugs in All Softmax Implementations(2026-10-27, 3 posts)
- Jon Barron: every softmax and cross-entropy implementation has two subtle numerical bugs — jon_barron · 2026-10-27
- Softmax's standard implementations hide two numerical bugs, Google paper shows — srush_nlp · 2026-10-28
- Same softmax numerical bug also affects KL regularizers in RL, researcher notes — heghbalz · 2026-10-28