SentZero: An Enhanced Sentence-Centric Vision-Language Pretraining for Multi-Task Zero-Shot Chest X-Ray Analysis
Hangyul Yoon, Hyungyung Lee, Edward Choi, Eunho Yang
cs.CV, cs.AI
2026-09-28
SentZero maps CXR phrases to abstract sentences and adds a patch-level FN loss, hitting 0.732 pointing on ChestXDet10, 7.6 points over the strongest prior.
Chest X-ray vision-language pretraining still usually needs a second, labeled fine-tune. Reports are long, dense, and writer-dependent, so alignment is noisier than CLIP-style natural captions. A prompt like "There is a dog" works for open-domain zero-shot. A short clinical prompt often fails to match how findings are written.
CARZero and RadZero already split reports into LLM-extracted clinical sentences and run classification plus grounding from one model. Radiology discourse adds a property those pipelines mostly skip. The finding vocabulary is small, so clinically equivalent lines such as "The lungs are clear" repeat across patients. Treating each extracted sentence as a caption of its own image wastes the report's semantic hierarchy (detail versus concept) and turns cross-study repeats into contrastive false negatives. Better phrase extraction does not remove them; the description space is intrinsically narrow.
SentZero is a KAIST sentence-centric pretrainer that transfers to multi-task CXR analysis with no task-specific fine-tuning. Three pieces.
Abstract-level mapping grows the positive set. Phrase extraction reuses RadZero's LLaMA3-70B-Instruct outputs, templated as "There [Presence] [Finding] of [Location]". Qwen3-Next-80B then compresses each phrase into a (Topic, Presence) tuple, Presence in {Yes, No, Maybe Yes}, and writes a short sentence such as "There is opacity". The original line "There is mild opacity in the bilateral lung base" and the abstract line are both positives. That gives two granularities of supervision and a handle for matching shared statements across studies.
The visual side is not a single [CLS]. Each sentence softmax-weights ViT patches by cosine similarity (temperature 0.1) and sums them. Two FiLM blocks (scale and shift predicted from a conditioning vector) then modulate that aggregate from the concatenated visual and sentence features, added back as a residual. The sentence edits how the image is read; the spatially pooled content stays. The backbone is RAD-DINO with LoRA (rank 16, alpha 48) plus four trainable transformer blocks, fusing layers 3, 6, and 9 with the last layer. Text is MPNet (all-mpnet-base-v2). Input is 518×518, 37×37 = 1369 patches.
False negatives keep their contrastive assignment. If sentence S of image j also sits in image i's augmented sentence set, (i, S) is a false negative. The method does not drop it from the InfoNCE denominator and does not relabel it as a positive. An auxiliary loss (weight 0.01) attracts only the top-20% most similar patches toward S, using 1 minus cosine. The global objective stays image-to-text MP-NCE (multiple positives per image, each competing only against other images' sentences) plus text-to-image InfoNCE.
Pretraining uses MIMIC-CXR frontal and lateral views: about 377K images, 227K studies, 65,379 patients, findings section only. AdamW, learning rate 1e-4, batch 256, up to 20 epochs on four NVIDIA H200 GPUs, roughly 5 hours, early stop on validation loss with patience 3. At inference, classification uses a "There is [Finding]" prompt, cosine similarity, and a sigmoid. Grounding upsamples the patch attention map and takes the peak.
Classification is AUROC. Grounding is pointing-game accuracy: the heatmap peak must fall inside the ground-truth box. Zero-shot baselines are GLoRIA, BioViL-T, MedKLIP, KAD, CARZero, RadZero, CoNNs, and GLINT (ViT-B, 512 input).
| Method | OpenI | CXR14 | PadChest | ChestXDet10 cls | CheXpert | ChestXDet10 point | MS-CXR |
| CARZero | 0.838 | 0.811 | 0.810 | 0.796 | 0.923 | 0.543 | 0.872 |
| RadZero | 0.847 | 0.804 | 0.841 | 0.787 | 0.900 | 0.622 | 0.886 |
| CoNNs | 0.871 | 0.819 | 0.835 | 0.825 | 0.920 | 0.656 | n/a |
| GLINT | 0.871 | 0.817 | 0.853 | 0.812 | 0.918 | 0.625 | n/a |
| SentZero | 0.889 | 0.839 | 0.861 | 0.843 | 0.904 | 0.732 | 0.922 |
On ChestXDet10 grounding SentZero scores 0.732 versus CoNNs at 0.656, a 7.6 point gap. On MS-CXR it scores 0.922 versus RadZero at 0.886, +3.6 points. It leads OpenI, ChestXray14, PadChest, and ChestXDet10 classification. CheXpert has 500 cases; CARZero's 0.923 beats SentZero's 0.904.
Ablations start from all four modules off. Multi-layer fusion mainly helps grounding (ChestXDet10 0.651→0.666, MS-CXR 0.892→0.922). Abstract mapping is the largest classification lift (ChestXDet10 classification 0.809→0.840). Text-conditioned residual alone is strongest on PadChest and CheXpert (CheXpert 0.916). The full stack gives those two sets back a little and pushes grounding to 0.732 / 0.922, winning the other five benchmarks by a wider margin than TC-only.
False-negative variants are sharper. With no FN handling, ChestXDet10 pointing is 0.712 and MS-CXR 0.904. Masking is mostly worse. Relabeling false negatives as positives collapses several sets: OpenI 0.886→0.849, PadChest 0.863→0.790, ChestXDet10 pointing 0.712→0.514, MS-CXR 0.904→0.707. The auxiliary loss keeps the original pairs and reaches 0.732 and 0.922. Editing the contrastive graph is harmful here.
For anyone building one CXR encoder for several zero-shot tasks, two takeaways. Report hierarchy and cross-patient repeats are usable supervision, not noise. Relabeling those repeats as positives is a bad fix; CoNNs-style relabel/mask mostly loses to leaving the pairs and pulling the relevant patches.
The training run is cheap: four H200s, about five hours. The real upfront cost is LLaMA3-70B phrase extraction plus Qwen3-Next-80B mapping. When boxes and disease labels are missing, prompt classification plus pointing is a practical path.
This is an increment on the RadZero sentence-level line, not a new paradigm. Backbone, phrase extraction, and multi-positive contrast are inherited. The reusable bits are abstract positives, FiLM residual readout, and the empirical warning against relabeling false negatives.
There is no dedicated limitations section. The numbers leave gaps.
CheXpert trails CARZero by 0.019 AUROC. The full model also sits below the TC-only ablation on PadChest and CheXpert. The paper blames CheXpert's 500 cases for the first gap and does not explain the second. The joint objective may trade classification on small sets for grounding; that split is not tested.
False negatives are detected by membership in the augmented sentence set, i.e. literal sharing after templating. Paraphrases that never collapse to the same Topic remain hard negatives. The paper shows that changing pair assignments hurts. It does not show that concept-level matching is useless, only that CoNNs' relabel/mask package is weaker in this implementation.
Pointing game credits a hit if the peak is inside the box. Large boxes are easy, and the score is silent on heatmap spread or multi-lesion cases. Figure 2 shows small-lesion examples. CoNNs and GLINT leave MS-CXR blank, so the 3.6 point claim is only versus RadZero and CARZero.
Pretraining is English MIMIC-CXR findings only. How Maybe Yes enters the loss is barely specified. Inference still uses "There is [Finding]". Non-English and non-templated reports are untested.