41000 measurements show scientific errors are heavy-tailed, 5σ outliers vastly exceed Gaussian odds

tetsuoai · x · 2026-11-21

tetsuoai proposes a reproducible exercise: take Bailey's 'Not Normal' dataset of human scientific measurements, plus machine-predictions-of-nature data, divide each miss by its stated error bar, and plot both shapes to see whether machines miss like scientists do.

The underlying paper analyzed 41,000 measurements of 3,200 quantities across medicine, nuclear and particle physics, and interlaboratory comparisons. Key findings: 5σ disagreements occur up to five orders of magnitude more often than naive Gaussian expectations; uncertainty-normalized differences follow heavy-tailed Student-t distributions often nearly Cauchy; errors from mistakes and unknown systematic problems show power-law distributions akin to failures in designed complex systems; physics uncertainty estimates improve much faster than medicine's.

Original post →

More from Research

Research channel →