Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation
Aditya Parikh, Aasa Feragen, Sneha Das, Stella Frank
cs.CL, cs.AI
2026-03-02
Radiology VLMs hit BERTScore 0.78 by collapsing to safe templates that erase sex-linked terms. CAD and WAE metrics expose the loss; stochastic decoding swaps erasure for new bias.
Radiology report generation uses vision-language models to turn chest X-rays into text, and the field grades these reports almost entirely on token overlap: BLEU, ROUGE, BERTScore. The more a generated report overlaps a clinician's reference, the higher it scores.
That setup invites template collapse. Models repeat safe templates such as 'no acute findings', and because normal films dominate the data, these lazy outputs score well. This paper from DTU targets a less obvious layer of the same failure. What gets erased is not only patient-specific detail but clinical terms bound to patient sex, and no standard metric notices the loss. This is a measurement and fairness paper, not a new model.
The authors first define what it means for a word to be associated with a sex. For each clinical term w they compute a Dirichlet-smoothed log ratio of unigram probabilities, s(w) = log[pF(w) / pM(w)], and convert it to a z-score. Mastectomy reads as strongly female in the references; AICD (an implanted defibrillator) reads as strongly male.
Clinical Association Displacement (CAD) tracks how each term's association shifts between reference and prediction, sorted into three buckets: Erasure (strong in reference, neutral in prediction), New Bias or Bias Flip (a fresh association appears or flips), and Preservation. Thresholds are zneutral = 1 and zstrong = 2.
Weighted Association Erasure (WAE) aggregates displacement across the whole vocabulary into one global signal-loss score, weighted by term frequency, with per-sex WAEg and a disparity delta WAE. Significance is confirmed by label-permutation tests.
The reason for a new family: existing fairness metrics use group-wise F1 gaps and miss vocabulary redistribution, while WEAT operates on static embeddings and is blind to shifts caused by decoding. CAD and WAE act on the actual generated vocabulary. The authors also treat decoding as a confounding variable and sweep six strategies: greedy and beam search on the deterministic side, and conservative, rich, top-k, and nucleus sampling on the stochastic side.
ReX-QWEN (Qwen3-VL-8B fine-tuned with LoRA on 60K films) beats its baselines on standard metrics but shows template diversity of only 14.1-52.7%. Across the six strategies, deterministic ones collapse to 22-38 unique terms, about 5% of the reference vocabulary, and greedy and beam produce only 0.3-0.5% unique reports.
| Decoding | BERTScore F1 | Erased terms | Template diversity | WAE (pred) |
| Greedy / beam | 0.78 | 32-35 | 0.3-0.5% | 2.50 |
| Rich (T=0.7) | slightly lower | fewer, but +25-65 new bias terms | 36.3-100% | 33.81 |
Greedy decoding still posts BERTScore F1 = 0.78, yet CAD flags 32-35 erased terms, covering nearly every strongly sex-bound term in the references. Mastectomy and AICD both drift toward neutral association.
Stochastic sampling restores the vocabulary at a cost: pneumothorax is neutral in the references (zref = 0.15) but low-temperature generation hallucinates a strong male association (zpred = -5.69). WEAT's effect size stays near zero throughout, confirming it cannot see decoding-induced shifts.
The sex-skewed training experiment is the sharpest result. Switching the training mix to 67% male or 67% female leaves every standard metric flat (BERTScore F1 still 0.76-0.78, RateScore 0.54-0.57, WEAT-ES near 0), while delta WAE moves from -0.05 for the balanced baseline to +0.79 for the male-skewed model and -0.94 for the female-skewed one. The pattern reproduces on LLaVA-Rad across ReXGradient and CheXpert: deterministic decoding gives 45.7% template diversity, 19 erased terms, 5 bias flips; rich sampling gives 91.1% diversity, 11 erased terms, and 12 bias flips.
An ablation sweeps the aggregation threshold across its full range with no change to the ranking of strategies, which marks the trade-off as a generation mechanic rather than a thresholding artifact.
For anyone deploying medical VLMs, the takeaway is direct: do not pick a model or decoding strategy on BERTScore and ROUGE alone. A model scoring 0.78 can silently drop the findings tied to one patient group, and your leaderboard will show nothing wrong. Adding CAD, WAE, and a couple of diversity metrics is cheap, and it surfaces a failure mode the current stack is built to miss.
Decoding is a fairness lever, not only a quality lever, and that holds beyond radiology. Any generation eval that rewards safe, generic output is gameable the same way.
It is an evaluation and diagnostic paper, not a new model. It will not hand you better reports; it tells you the 'good' you are measuring is wrong.
The authors concede that the analysis covers binary sex only, with age, ethnicity, and intersectional groups left to future work; that clinician review is needed to confirm whether CAD-flagged erasures map to clinically meaningful omissions; and that the study is limited to chest X-rays.
Two further gaps stand out. The ablation sweeps the aggregation threshold but not the word-level thresholds zneutral and zstrong themselves. Whether new bias terms are recovered signal or hallucinated noise is posed as an open question and never resolved; WAE flags displacement but cannot separate a true recovered term from a fabricated one. The N=2,000 evaluation subset is small for vocabulary-level association statistics on rare terms, and a zero-shot GPT-4.1 baseline is a slightly unfair foil for fine-tuned medical models.