2026-09-29
A review of 184 high-impact biomedical AI papers: 97% used invalid CV tests. Of 68 re-analyses, 69% lost significance; about one in three abstracts rest on a spurious result.
Biomedical papers increasingly pick algorithms and biomarkers by who wins a predictive bake-off. The usual bake-off is K-fold cross-validation: split the data into K folds, rotate which fold is the test set. Test folds do not overlap. Training folds do, heavily, and every test fold later re-enters training. Fold-level performance differences are therefore positively correlated.
Paired t-tests and Wilcoxon tests treat those K numbers as independent. Variance is too small, false positives too high. Dietterich (1998) and Nadeau and Bengio (2003) already described the failure. How common it is in biomedical AI, and how many published "significant" wins collapse once the correlation is counted, had not been measured in high-impact journals.
Yeo's group at NUS and Thomas Nichols at Oxford ran a PRISMA search on PubMed for original articles from 1 June 2020 to 1 June 2025 in journals with impact factor at least 15 that compared predictive performance with cross-validation. Of 2,400 records, 1,875 compared models; 1,008 of those (54%) neither applied a clear statistical test nor reported confidence intervals. The final set is 184 papers where the test could be classified as ignoring or accounting for between-fold correlation. Screening used Claude; two human raters independently verified the decisive labels.
For 68 papers with fold-level statistics, or with exact p-values and degrees of freedom that reconstruct them, originally significant comparisons were re-run with Nadeau and Bengio's corrected resampled t-test. A comparison that then had p ≥ 0.05 is called spurious. SHARP cannot be applied after the fact because it changes the cross-validation design.
SHARP (Split-Half Analysis of Repeated Performance) randomly cuts the dataset into two disjoint halves, runs K-fold cross-validation inside each half, and repeats this J times. The two halves in a given repeat are independent; repeats still share data, so they remain correlated. That geometry lets variance and correlation be estimated from the data instead of being fixed in advance. Inference uses a score test, which controlled false positives best in their simulations. Main results use SHARP5-R (5-fold, 60 repeats) and SHARP10-R (10-fold, 30 repeats).
Simulations span EMNIST digits, UK Biobank gray-matter volumes, Covertype land cover, and KEGG metabolic pathways, sample sizes 100 to 2,000, and three hyperparameter regimes (fixed, one inner split, nested CV): 420 scenarios, scored against 12 other tests on false-positive rate, power, and interval coverage.
178 of 184 studies (97%) used invalid tests that assume fold independence: 95% among classical ML, 97% among deep nets, 100% among papers that used both. Uncorrected t-tests and Wilcoxon tests were the usual choices. Paper volume grew 14.5% per year; the invalid-test share did not fall (permutation p = 0.63). Among seven fields with at least 10 papers, neuroscience was lowest at 82%; genetics, biotechnology, and biochemistry sat at 100%. No link showed up with journal impact factor, a 0–14 rigor score, or whether data or code was shared.
Of 68 re-analyzable studies, 47 (69%) contained a spurious comparison. Of 37 with an abstract claim backed by an invalid test, 22 (59%) had a supporting comparison that failed. Across those 47, the median of each study's largest corrected p-value was 0.23; the median of within-study median spurious p-values was 0.15; for the 22 abstract cases it was 0.17. p = 0.17 is |z| = 1.37 versus 1.96 at p = 0.05, so significance would need about a 43% larger effect. Extrapolating the 69% rate to 173 at-risk studies gives (173/184) × (47/68) = 65% with a spurious comparison, and (100/184) × (22/37) = 32% with an abstract claim sitting on one.
Under a constructed null, one run of 10-fold CV gave paired t and Wilcoxon false-positive rates near 19%; in 64% of scenarios the 95% CI sat entirely above 0.05. After 30 repeats the rate was 77%, inflated in every scenario, and it approaches 100% as repeats grow. A finite sample always shows some difference; independence pretends the sample size grows with each repeat, so uncertainty shrinks until anything looks significant. On KEGG, switching from fold-averaged to sample-level statistics lifted one-run 10-fold FPR from 24% to 38%.
SHARP held false positives near the nominal 5%, matching the 5×2 paired t-test and the empirical test of differences. The corrected resampled t-test broke the 5% bound in 19% of scenarios under 5-fold CV and 44% under 10-fold, and that control depended on the simulation scheme. SHARP5-R had the highest power, 27.8 percentage points above Dietterich's 5×2 paired t-test, and won every pairwise comparison. Mean 95% CI coverage was 96% for SHARP5-R and SHARP10-R. Independence-based intervals were too narrow; 5×2 and the empirical test were too wide, with coverage near 100%.
Point estimates from cross-validation remain valid estimates of generalization to the sampled population. What fails is the uncertainty: p-values, intervals, and "A beats B" headlines. Repeating CV for stability, then feeding the extra folds into an independence test, is the fastest way to a 100% false-positive rate.
For a single dataset the authors default to SHARP, and to the corrected resampled t-test when a split-half would starve training. Wilcoxon across datasets is for six or more independent datasets, not for folds inside one. External validation asks whether a fitted model travels; it does not induce fold correlation, so it should not reuse the CV test. The paper lists a reporting floor: number of folds, number of repeats, whether splits were shared, the named test, and the shape of the vector that entered it. They describe a scikit-learn-compatible implementation; the GitHub URL in the preprint is still a placeholder.
The statistical bug is old. Measuring how often high-impact biomedical AI commits it, and shipping a test that estimates the correlation instead of guessing it, is the contribution.
The 65% and 32% figures extrapolate from 68 papers to 173, then to the wider literature. The authors argue the 68 are not the safer subset (re-analysis needed complete reporting), that some unreanalyzed tests such as DeLong inflate false positives even more, that the corrected t-test itself is not always calibrated, and that they mostly skipped extra multiple-comparison penalties. Impact factor was already ≥15 and still uncorrelated with invalid testing. That package of caveats points toward "conservative." It is still an extrapolation.
SHARP trains on half the data. If two algorithms scale differently with sample size, the ranking inside SHARP can diverge from full-data CV. Ten-class EMNIST at N = 100 was dropped because split-half plus folds could not guarantee a class in every test fold. Simulations used logistic regression, ridge, and Extra Trees, not large neural nets. The GitHub link is unfilled. A spurious comparison does not prove the scientific claim is false; it proves the reported evidence was weaker. The paper is a September 2026 preprint.