Subtle numerical bugs in softmax/cross-entropy may cause mysterious training loss spikes
lmoroney · x · 2026-10-24
Citing Jon Barron's result, lmoroney explains two subtle numerical bugs in standard softmax and cross-entropy implementations (JAX, PyTorch, even textbooks): "Softmax Collapse", where one dominant logit's gradient rounds to zero while others don't so gradients no longer sum to zero; and "Softmax Cancellation", where near-perfect outputs (e.g. a student initialized as the teacher in distillation) still produce non-zero gradients. A simple rewrite fixes both and removes the "slingshot" spikes seen during grokking. He urges learners to understand the math under the APIs and try the fix on a small model.
More from Research
- Paper: "bottleneck pressure" metric shows where AI-infeasible work gains value as AI speeds up the rest — soumitrashukla9 · 2026-10-28
- New IEEE IT paper: geometric Jensen-Shannon divergence between Gaussian measures on Hilbert space — FrnkNlsn · 2026-10-28
- CPR-Dataset: retrievers fail at cross-domain discovery, training gives +70% on BRIGHT — coallaoh · 2026-10-28
- Hugging Face robot duck walks in simulated snow with 290,000 physics-simulated flakes — lukas_m_ziegler · 2026-10-28
- Mercor, Stanford and NBER launch TaskNet, an open-source task catalog updating O*NET for the AI era — soumitrashukla9 · 2026-10-28
- François Fleuret Proposes Tagging Every Token 'Heard/Thought/Said' to Solve Alignment — francoisfleuret · 2026-10-28