DeepMind's Jon Barron: every softmax and cross-entropy implementation has two subtle bugs
jon_barron · x · 2026-10-29
Jon Barron (Google DeepMind) shares a new finding: all existing implementations of softmax and cross-entropy — including JAX, PyTorch, and textbooks — contain two subtle bugs. A simple rewrite fixes both and eliminates "slingshots" during grokking. DeepMind research lead Raia Hadsell amplified it as "a good reminder to question everything."
More from Research
- Hexels: JHU's Hexagonal Primitives Cut VRAM 5.5x While Beating 3DGS by 4.96dB — ssh4net · 2026-10-29
- 3,957-task study finds no reliable way to combine human and AI oversight yet — sethlazar · 2026-10-29
- New preprint measures 'articulation gaps': how much knowledge in humans and AI is tacit — sanmikoyejo · 2026-10-29
- Peer Review Is the Least Articulable Field: Five Findings on Tacit Knowledge — sanmikoyejo · 2026-10-29
- New Prompt Optimization Beats GEPA by ~14% and Estimates Performance Bounds — sanmikoyejo · 2026-10-29
- Borrowing Capture-Recapture from Ecology to Bound Rubric Performance — sanmikoyejo · 2026-10-29