Jon Barron: every softmax/cross-entropy implementation in JAX, PyTorch has two subtle bugs
khademinori · x · 2026-10-24
Jon Barron shared a new finding: all existing implementations of softmax and cross-entropy — including JAX, PyTorch, and textbooks — contain two subtle bugs. A simple rewrite fixes both and eliminates "slingshots" during grokking. Researchers including BlackHC amplified the result, noting how surprising it is that "basic" ML operations still have room for improvement.
More from Research
- Paper: "bottleneck pressure" metric shows where AI-infeasible work gains value as AI speeds up the rest — soumitrashukla9 · 2026-10-28
- New IEEE IT paper: geometric Jensen-Shannon divergence between Gaussian measures on Hilbert space — FrnkNlsn · 2026-10-28
- One-page Q-learning notes with math, code and a playground for RL beginners — East-Muffin-6472 · 2026-10-28
- CPR-Dataset: retrievers fail at cross-domain discovery, training gives +70% on BRIGHT — coallaoh · 2026-10-28
- Hugging Face robot duck walks in simulated snow with 290,000 physics-simulated flakes — lukas_m_ziegler · 2026-10-28
- Mercor, Stanford and NBER launch TaskNet, an open-source task catalog updating O*NET for the AI era — soumitrashukla9 · 2026-10-28