Far from training data, ReLU classifiers grow more confident with every step — a 2019 theorem explains why softmax can't say 'none of these'
bravo_abad · x · 2026-11-23
Jorge Bravo Abad explains a 2019 theorem by Hein, Andriushchenko and Bitterwolf: ReLU classifiers, given inputs not confined to a fixed range, see confidence tend to 100% far from training data.
Why
- Softmax compares logits exponentially: a gap of 4.6 already means 99%, and each extra 2.3 cuts remaining doubt tenfold
- ReLU networks are piecewise linear; far from data each class score becomes a straight line, gaps grow with distance, so odds grow exponentially
Consequences
- Softmax answers 'which known class fits best', never 'does this input resemble anything I trained on' — it cannot say 'none of these'
- Example: a collider-event classifier trained on simulations must file genuinely novel processes under known labels, most confidently for the most extreme events — exactly the discoveries you hope to find
- Post-hoc calibration can't fix it since the lead grows without bound; familiarity must be measured separately, via input-density models or classifiers designed to lose confidence off-distribution
Softmax ranks the known answers; it can't tell you the question is new.
More from Research
- The New Yorker: A.I. Is Solving Decades-Old Math Problems, With Terence Tao as Guide — EricTopol · 2026-11-23
- Mathematician proposes making local models great at Lean theorem proving — StefanoGogioso · 2026-11-23
- LLM Study of 1,298 Scenarios Finds Models Blame Men 23 Points More for Identical Misconduct — alex_verem · 2026-11-23
- 55-Page Riemann Hypothesis Proof Guide, De-Formalized from Lean with Gemini and GPT — chaumian · 2026-11-23
- Papers on AI timelines are growing super-exponentially, now doubling every 3.8 months — peterwildeford · 2026-11-23
- Unreleased frontier AI formalizes Fermat's Last Theorem proof, fells Millennium Problems, roils mathematicians — Afinetheorem · 2026-11-23