Jon Barron: every softmax and cross-entropy implementation (JAX, PyTorch) has two subtle bugs
IgorCarron · x · 2026-10-24
Google researcher Jon Barron shared a new result: all existing implementations of softmax and cross-entropy — including JAX, PyTorch, and textbooks — contain two subtle bugs. A simple rewrite fixes both and eliminates "slingshots" during grokking.
Key points:
- Extremely broad impact: every mainstream DL framework and standard textbook implementation is affected;
- The bugs are subtle and mostly harmless in ordinary training, but they trigger slingshot-style loss spikes in dynamics-sensitive regimes like grokking;
- The fix is a simple rewrite of the computation, with paper and code released.
A rare foundational numerical issue relevant to anyone training models at scale.
More from Research
- Paper: "bottleneck pressure" metric shows where AI-infeasible work gains value as AI speeds up the rest — soumitrashukla9 · 2026-10-28
- New IEEE IT paper: geometric Jensen-Shannon divergence between Gaussian measures on Hilbert space — FrnkNlsn · 2026-10-28
- One-page Q-learning notes with math, code and a playground for RL beginners — East-Muffin-6472 · 2026-10-28
- CPR-Dataset: retrievers fail at cross-domain discovery, training gives +70% on BRIGHT — coallaoh · 2026-10-28
- Hugging Face robot duck walks in simulated snow with 290,000 physics-simulated flakes — lukas_m_ziegler · 2026-10-28
- Mercor, Stanford and NBER launch TaskNet, an open-source task catalog updating O*NET for the AI era — soumitrashukla9 · 2026-10-28