Replication study shows sampling variation alone cuts estimates 12% and flips 9% of results
RexDouglass · x · 2026-09-04
- The I4Replication thread argues that sampling variation in the data itself is a quantitatively important driver of fragile empirical results.
- Key findings from their resampling analysis:
- Point estimates fall 12% on average; standard errors rise 40%.
- 70% of resamples yield a smaller t-statistic than the original (vs. 50% expected if published results were representative draws).
- 9% of results flip sign, and over half of those flips are themselves significant at the 5% level.
- Takeaway: published estimates may not be representative draws from the underlying population, and sampling noise alone can materially destabilize empirical conclusions.
Related event: Resampling study shakes significance of top-journal findings(3 posts)→
More from Research
- Machine Translation Is Not Solved: Last Translation Benchmark Released — zouharvi · 2026-09-04
- Is recurrent depth the real driver? Researchers debate 3x depth vs 81x params math — xuanalogue · 2026-09-04
- Causal Foundation Models: Pretrained Nets Estimate Treatment Effects via In-Context Learning — chaumian · 2026-09-04
- Indo-German workshop on Green AI for healthcare set for IIT Delhi — Tanmoy_Chak · 2026-09-04
- A nightwatchman on every probe: superintelligent surveillance vs galactic anarchy — jachiam0 · 2026-09-04
- Epoch AI launches FrontierMath Erdős benchmark of 68 unsolved problems; Astra tops it at 2/68 — keviv9 · 2026-09-04