Jon Barron: every softmax & cross-entropy implementation (JAX, PyTorch) has two subtle bugs — a simple rewrite fixes both and removes grokking slingshots
chrisemoody · x · 2026-10-25
Jon Barron shared a new result claiming that all existing implementations of softmax and cross-entropy — in JAX, PyTorch, and even textbooks — contain two subtle bugs. A simple rewrite fixes both and eliminates the "slingshot" spikes observed during grokking.
Prof. JB Huang of UT Austin enjoyed the paper enough to write an accessible explainer thread spreading the findings.
More from Research
- Paper: "bottleneck pressure" metric shows where AI-infeasible work gains value as AI speeds up the rest — soumitrashukla9 · 2026-10-28
- New IEEE IT paper: geometric Jensen-Shannon divergence between Gaussian measures on Hilbert space — FrnkNlsn · 2026-10-28
- One-page Q-learning notes with math, code and a playground for RL beginners — East-Muffin-6472 · 2026-10-28
- CPR-Dataset: retrievers fail at cross-domain discovery, training gives +70% on BRIGHT — coallaoh · 2026-10-28
- Hugging Face robot duck walks in simulated snow with 290,000 physics-simulated flakes — lukas_m_ziegler · 2026-10-28
- Mercor, Stanford and NBER launch TaskNet, an open-source task catalog updating O*NET for the AI era — soumitrashukla9 · 2026-10-28