41000 measurements show scientific errors are heavy-tailed, 5σ outliers vastly exceed Gaussian odds
tetsuoai · x · 2026-11-21
tetsuoai proposes a reproducible exercise: take Bailey's 'Not Normal' dataset of human scientific measurements, plus machine-predictions-of-nature data, divide each miss by its stated error bar, and plot both shapes to see whether machines miss like scientists do.
The underlying paper analyzed 41,000 measurements of 3,200 quantities across medicine, nuclear and particle physics, and interlaboratory comparisons. Key findings: 5σ disagreements occur up to five orders of magnitude more often than naive Gaussian expectations; uncertainty-normalized differences follow heavy-tailed Student-t distributions often nearly Cauchy; errors from mistakes and unknown systematic problems show power-law distributions akin to failures in designed complex systems; physics uncertainty estimates improve much faster than medicine's.
More from Research
- Riemann Hypothesis proof gets full formal verification, experts call it 'very human math' — Kyrannio · 2026-11-21
- Hassabis predicted 2 years for AI to crack a Millennium Prize problem; 6 months later, both Riemann and Navier–Stokes have proposed solutions — Hesamation · 2026-11-21
- GPT-2 'sleep' replay from a frozen snapshot keeps 58% of old facts vs 6% without — Junior_Reputation639 · 2026-11-21
- ERC reviewer rejects LLM autoformalization proposal: "tried without good results" — marc_lelarge · 2026-11-21
- Terence Tao on AI math: 'strip mining' that wins proofs but teaches us nothing — yacineMTB · 2026-11-21
- Anthropic claims Claude proved the Riemann hypothesis; Stanford posts all-human rival proof hours later — The Decoder · 2026-11-21