ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs
Sebastián Andrés Cajas Ordóñez, Maximin Lange, Quang Bui, Anqi Peter Li, Felipe Ocampo Osorio, Rafi Al Attrach, Kushul Reddy Palakala, Sahil Kapadia, Zakaria Laouabdia Sellami, Xinyue Zhang, Ashley Zhang, Leo Anthony Celi
cs.CV, cs.AI
2026-09-14
ModaLens swaps the chest X-ray with question and report held fixed. MedGemma-27B flips on 4.26% of 44,786 trials with the report and 20.94% without it (+16.7 points).
Medical vision-language models often see a chest radiograph and its radiology report together. If the report already contains the answer, a correct output does not mean the model used the image. Ignoring the image is a known failure in general VQA. In radiology it is worse: the report may describe a prior study, be wrong, or name a finding that has since resolved. A forced yes/no gives no signal when image and report agree.
ModaLens does not score visual correctness. It measures a cleaner quantity: hold the question and the report fixed, replace the entire image with one from another study, and count whether the answer moves.
The primary cohort is the MIMIC-CXR test split: 293 patients, 3,199 cases (one frontal image and one substitute), 14 questions per case (13 CheXpert findings plus one composite about any acute cardiopulmonary finding), 44,786 paired trials. The substitute is drawn from the same patient's other frontal studies when possible, preferring a different CheXpert-positive set; 19 single-study patients use another patient. 3,180 pairs are within-patient, 19 are not.
The main model is MedGemma-27B, an instruction-tuned Gemma 3 with 62 decoder layers. One user turn: an image block, then Report: and Question:. The primary readout is whether the generated Yes/No flips after an explicit answer instruction. The paper's first measurement, a greedy argmax over lowercase yes/no at the first generated token, is kept as a sensitivity analysis. Intervals are patient-clustered bootstraps.
Controls include dropping the report or the image, swapping text-block order, warning that the report may be wrong, replacing the report with neutral prose or clinical boilerplate, cumulatively masking attention to report tokens from a chosen layer, and repeating the protocol on Qwen3.5-9B, Qwen3.5-27B, and LLaVA-NeXT on Mistral-7B.
Under the explicit answer instruction, generated-answer flip rates are:
| Condition | Flip rate |
| Report present | 4.26% [3.95, 4.60] |
| Report absent | 20.94% [19.70, 22.13] |
| Paired difference | +16.7 points [15.6, 17.7] |
The original prompt with the lowercase first-token readout gives 4.70% versus 17.07%, a 12.4-point gap. Mean absolute change in the continuous margin \(\log p(\texttt{yes})-\log p(\texttt{no})\) is 0.690 with the report and 2.307 without it; 82.9% of trials move more without the report. Only 4.7% flip with the report present, yet 32.2% move by more than 0.5 in margin. A stable binary answer is not a frozen image channel.
The pooled gap is not uniform across findings. Without the report, atelectasis and support devices are answered yes on 97.8% to 100% of trials, and their flip rates fall: 6.81% with the report versus 4.63% without for atelectasis, 11.03% versus 1.44% for support devices, under the generated answer. The other nine findings run 16.8% to 32.0% without the report.
Labels come from report text. With the report present, flip rates are 4.30% on label-changed trials and 4.81% on label-unchanged trials. Scored against each study's own report-derived label, accuracy is 0.648 on the original image and 0.589 on the substitute. Modality ablation: report alone has balanced accuracy 0.770; adding the image drops it to 0.717; image plus question is 0.665. The image adds 0.165 without the report and subtracts 0.053 with it.
On the one-question design, length-matched neutral prose flips 8.2% and clinical boilerplate 7.1%, against 10.3% with no report and 1.4% with the full report. Dropping sentences that mention the queried finding raises the rate to 11.6%; a sentence stating the reference answer lowers it to 3.1%. Warning that the report may be wrong moves the one-question rate from 1.22% to 1.50%, a 0.28-point gap whose interval includes zero.
The paired report effect is 13.3 points on Qwen3.5-9B, 19.60 on Qwen3.5-27B, and 12.75 on LLaVA-NeXT. Masking attention to report tokens from layer 0 raises the one-question flip rate from 1.50% to 7.75% (12.00% with no report); blocking from layer 32 leaves the baseline. Single-layer blocks never exceed 2.50%.
Anyone putting a report in context for a chest-X-ray VLM can reuse this audit: run the same case twice, swap only the image. 4.26% versus 20.94% says MedGemma-27B's Yes/No barely tracks the image once the report is present. That is not automatically a clinical error, because a report is often an expert reading. If the task is to read this image, the report suppresses image sensitivity, and a one-line warning does not restore it. Measure the report-present and report-absent arms separately. Do not treat report-conditioned accuracy as visual competence.
Labels are report-derived throughout; there is no independent visual gold standard. Substitutes are unmatched on view and on the other thirteen labels, so a swap can keep the queried finding and depress the flip rate. Magnitude depends on text-block order: +12.4 points with the image first, +7.8 with the text first. Every report-availability estimate is one hospital's English MIMIC-CXR reports. Several findings are near-constant yes without the report, so the pooled 16.7 points is not a per-finding law. Readout choice changes the report-absent absolute rate; the direction does not.