Can saliency maps flag chest radiograph classifier errors at a new hospital? An external validation study
Aashish Chapagai, Angad Singh Gurtatta, Erik Cambria
cs.CV, cs.LG, eess.IV
2026-11-03
Across 303,768 chest X-ray decisions at US and Vietnamese hospitals, saliency concentration added ≤0.011 AUC to confidence-based detection and reversed direction between sites.
Chest X-ray classifiers trained on large hospital datasets read about as well as radiologists on held-out data from the same hospital, then degrade when moved elsewhere, partly because they learn acquisition- and site-specific shortcuts. Many of the errors are silent: a wrong output looks exactly as confident as a correct one. The accepted safeguard is a saliency map beside each prediction, a heatmap of where the model looked, on the assumption that a reader will notice when it points at the wrong place; international consensus guidance lists explainability among its six principles for trustworthy medical AI. The evidence behind that assumption is thin. Maps localize worse than radiologists, some barely change when the model's weights are randomized, and a plausible map says nothing about generalization. Prior studies asked whether a map looks right. A deployed system needs something narrower: a label-free signal that a particular prediction is probably wrong, the failure-detection problem. Whether explanations contribute to it after a hospital move had never been tested.
NIH test patients were split into a calibration half and an evaluation half, and the failure-detection combination (a logistic model over margin and TTA) was fitted only on the calibration half. That mimics what a hospital can do before deployment, so the external numbers test exactly what a transferred check would face.
| Comparison | Metric | Result |
| Grad-CAM vs radiologists (CheXpert benchmark, 520 findings) | Localization hit rate | DenseNet 0.575 / ViT 0.381 / radiologists 0.888 |
| Margin alone | Failure-detection AUROCf | 0.646-0.685, best single score in 5 of 6 settings |
| Enrichment E alone | AUROCf (NIH) | 0.484 / 0.524, at chance |
| E added to margin+TTA | ΔAUROCf | Upper bound ≤0.011 at all sites, both models |
| DenseNet, VinDr-CXR | AURC (lower is better) | 0.427 with E vs 0.415 without |
The need is real: flag precision ranged from 0.343 to 0.529 across models and sites, so 47-66% of flags were false positives. Rankings transferred and operating points did not: at CheXpert, DenseNet-121 edema sensitivity fell from 0.424 to 0.008 and cardiomegaly from 0.409 to 0.136 while both findings kept AUROCs above 0.82; ViT-B/16 detected no edema at all. At VinDr-CXR the AUROC rose above 0.886 for both models, which the authors attribute to cleaner consensus labels than NIH's report-mined ones.
The reversal is the instructive part. The NIH-fitted combination gave enrichment a standardized weight of −0.071, meaning more concentrated maps were slightly more often wrong at equal confidence. At VinDr-CXR the relation ran the other way (E alone reached AUROCf 0.618), so the transferred detector hurt. Refitting with local labels flipped the weight positive (+0.348) and still did not help (ΔAUROCf +0.0091, interval spanning zero). Practically, keeping the half of DenseNet flags with the largest margin cut the VinDr-CXR false-flag rate from 0.566 to 0.421; selecting by enrichment only reached 0.503. On missed findings, enrichment was informative in reverse (AUROCf 0.221-0.351; near misses leave concentrated subthreshold evidence), and the margin already captures that (0.697-0.842). Sanity checks held: randomizing the classification head decorrelated Grad-CAM from trained maps, and LIME and SHAP reproduced the reversal on subsets, so the finding is not an artifact of one explainer.
Three deployable rules. Monitor with confidence, not heatmaps: keeping the more confident half of flags removed 6-14 percentage points of false flags at every site. Retune thresholds locally: ranking survives the hospital move, operating points do not. And an explanation-based check calibrated before deployment can hurt after it, which no single-site study could reveal. The null is bounded: gains above 0.011 AUROCf are excluded for a pre-deployment-calibrated detector, and above 0.024 even after local recalibration. There is also a structural argument: for a global-average-pooling head with a linear layer, Grad-CAM reduces exactly to the positive part of the class activation map, which sums to the class score minus its bias, so the map is a spatial rearrangement of evidence the score already summarizes rather than an independent measurement. The mechanism is plain. A confident false positive is produced by strong lesion evidence, and that evidence sits in the chest just as it does for a true positive; concentration cannot tell them apart.
Author-stated: NIH labels are report-mined and noisy; some external findings had very few positives (CheXpert lesion 10, pneumothorax 16); only one explanation summary, spatial concentration, was studied, while disagreement between explainers was large and other statistics might behave differently; no transformer-specific attribution for ViT; LIME and SHAP ran on at most 500 images per site; the site-refitted analyses were exploratory and post hoc; there was no reader study, so how radiologists would use heatmaps at a new hospital is unknown. Two more boundaries: the evidence covers one modality, two architectures and three sites, and other modalities are untested. The missed-finding refits did gain +0.024 to +0.052 locally, so explanation signals are not empty, just site-specific, and the same labeled cases are better spent re-tuning thresholds.