Softmax in 2 lines: why subtracting the max prevents overflow
cneuralnetwork · x · 2026-10-29
A concise explainer on the numerically stable softmax trick: e^x overflows to inf/NaN for large logits, but subtracting a constant from every logit leaves softmax unchanged since the e^(-c) factor cancels. Choosing c = max(x) makes the largest exponent e^0 = 1 and all others ≤ 1. Includes a two-line PyTorch implementation.
More from Research
- Axiom's agents are formalizing CFSG in 43M lines of code, possibly the largest open-source project ever — SucceededMind · 2026-10-29
- MIT's 'AI telepathy' KV-cache transfer beats text by 40% accuracy at 4x speed, across model vendors — LucaAmb · 2026-10-29
- Coding agents fail to reimplement models from papers, sparking doubts about how papers are written — charlieharris01 · 2026-10-29
- Information-theoretic scaling laws for diffusion models predict when memorization begins — davidad · 2026-10-29
- Amandine Flachs launches side project orbench — IgorCarron · 2026-10-29
- Critic says closed weights hide personas from users, but that doesn't prove models aren't general-purpose actors — aiamblichus · 2026-10-29