Jon Barron: every softmax & cross-entropy implementation (JAX, PyTorch) has two subtle bugs
code_star · x · 2026-10-24
Google researcher Jon Barron shares a new result: all existing implementations of softmax and cross-entropy — JAX, PyTorch, textbooks included — contain two subtle bugs. A simple rewrite fixes both and eliminates 'slingshots' during grokking. Commenters quip it will soon be a standard MLE interview question.
More from Research
- SenseTime open-sources 27B robot model matching GPT-6-Astra at ~1/50 the cost — rohanpaul_ai · 2026-10-24
- Cohere Labs brings five main track papers to EMNLP 2026 in Budapest — Cohere_Labs · 2026-10-24
- Gradient clipping stress test: unclipped RNN diverges in 17 steps, clipped runs converge to 0.043 loss — cneuralnetwork · 2026-10-24
- Reddit asks: why no folding@home-style crowdsourced open training dataset? — a300a300 · 2026-10-24
- AI paper reproduction failures trace mostly to the papers, not the AI agents — RexDouglass · 2026-10-24
- Agents reproduce most published findings: ~80-90% correct signs, >70% within CI — RexDouglass · 2026-10-24