Jon Barron: every softmax/cross-entropy implementation has two subtle numerical bugs
jon_barron · x · 2026-10-24
Google researcher Jon Barron announces that all existing implementations of softmax and cross-entropy — including JAX, PyTorch, and textbook versions — contain two subtle numerical bugs. A simple rewrite fixes both and eliminates the "slingshot" phenomenon observed during grokking.
More from Research
- UBC NLP lands 21 papers at EMNLP 2026, incl. 10 main conference — mageed · 2026-10-24
- UBC NLP lands 21 papers at EMNLP 2026 spanning culture, RAG, and agents — mageed · 2026-10-24
- Terminal-Bench exposes NVIDIA Nemotron 3.5 near-zero score vs Kimi K3.1's 65.7% — JJitsev · 2026-10-24
- Kimi K3.1 Pro reportedly first Chinese model to solve 20x20 nonograms — fairydreaming · 2026-10-24
- Self-Evolving Benchmarks: Multi-Agent Evolutionary Search Auto-Generates Challenging Tool-Use Tasks — MichaelD1729 · 2026-10-24
- Colossal didn't de-extinct the dire wolf — they gene-edited gray wolves to look like them — shae_mcl · 2026-10-24