FleXray: Universal Clinical X-ray Segmentation
Victor Ion Butoi, Vivek Gopalakrishnan, John V. Guttag, Adrian V. Dalca, Neel Dey
cs.CV, cs.AI
2026-09-23
FleXray learns 60-structure X-ray segmentation from CT projections plus Flux.2 edits, reaching mean Dice 0.875 on eight held-out datasets, 0.039 behind in-domain nnU-Nets.
CT and MRI already have generalist segmentors such as TotalSegmentator that outline a hundred structures from one volume. X-ray exams outnumber CT by a wide margin, yet almost none of that volume is quantitative. Projection collapses overlapping organs onto a 2D detector, so even experts struggle to draw dense pixel labels. Twelve public X-ray segmentation datasets hold 667,380 images, but the labels cluster on chest heart-and-lung maps.
The shortage is labels that cover the whole body, many projection geometries, and messy clinical appearance. Without those labels, Cobb angles, intraoperative navigation, and retrospective biomarkers still depend on a ruler and a radiologist.
FleXray is not trained on hand-labeled X-rays. A data engine takes six CT datasets (2,159 subjects; MOOSE supplies 1,597 densely labeled whole-body scans) and renders digitally reconstructed radiographs (DRRs) on the fly: ray integrals through the CT, with every 3D organ mask projected independently. A pixel can belong to several structures at once, so supervision is multi-label, not a mutually exclusive partition.
Each training step samples an anatomical target, viewpoint, field of view, magnification, and cone versus orthographic geometry, then multiplies every organ by its own attenuation factor to mimic density, contrast agents, and pathology. Hands and feet are rare in public CT, so four real X-ray sets (764 images) fill the gap: public hand and foot labels plus new forearm and humerus masks on MURA. Native ontologies are mapped into 60 classes: skull, sternum, sacrum, hips, scapulae, clavicles, limb bones, twelve rib pairs, each vertebra from C1 to L5, plus lungs, heart, liver, spleen, and kidneys. Brain and skeletal muscle, which X-ray does not show, are dropped. Left and right instances share a channel.
Analytic DRRs cover anatomy and geometry. They do not cover scatter, dose, detector response, or burned-in text. An off-the-shelf Flux.2-klein-9b editor is prompted to make MOOSE projections look like clinical films while keeping projected anatomy fixed. Flux sometimes rewrites anatomy on unusual views, so a U-Net trained only on raw DRRs filters pairs whose SoftDice drops too far. About 4% of enhanced renders are discarded.
Partial labels are handled honestly: unlabeled channels are ignored, not treated as absent; a hand film can still mark foot classes as true negatives. Loss is an equal mix of Soft-Dice and BCE. The network is a from-scratch seven-level 2D U-Net (widths 64 to 1024) with 60 independent sigmoids. The released system ensembles five members that differ only in the mix of online DRRs versus Flux-enhanced DRRs, then applies 16-sample test-time augmentation. Inference needs neither prompts nor acquisition metadata. Training uses 256×256 inputs for 2,000 epochs.
On eight held-out real X-ray datasets (wrist, lower limb, lateral elbow, pelvis, intraoperative fluoro, pediatric torso, ribs, chest), FleXray's equal-dataset mean Dice is 0.875. It beats PAXray, TotalSegmentator2D, and FluoroSAM on seven datasets (p much less than 0.001 on six). On VinDr-Rib it is statistically tied with the chest-specific PAXray (p=0.146). Dataset-specific nnU-Nets average 0.914, so FleXray trails by 0.039. On DeepFluoro it beats the in-domain nnU-Net, 0.915 versus 0.866, because that set has few training frames and a wide angle range.
| Method | Equal-dataset Dice | Setup |
| FluoroSAM | 0.348 | Ground-truth text prompt and click per class |
| FleXray | 0.875 | No prompts, no acquisition metadata |
| In-domain nnU-Net | 0.914 | One network per dataset |
On 213 DeepFluoro frames, Dice is 0.925 at 5–10° from frontal and 0.888 at 35–40°. Femur, hip, and sacrum stay within 0.05 of frontal performance; lumbar spine falls from 0.840 within 15° to 0.751 beyond 25°. FluoroSAM and TotalSegmentator2D average 0.711 and 0.371.
Flux editing leaves the overall mean unchanged (0.870 vs 0.870 on the single model) but lifts ribs 5–10 on VinDr-Rib by 3.9 Dice points, and rib 10 by 7.4, where analytic DRRs bury lower ribs under abdominal organs. Attenuation randomization raises pneumonia lung Dice on DarwinCVD19 from 0.894 to 0.914 (n=711) and barely moves healthy lungs (0.945 to 0.946, n=241). Pneumonia images below 0.85 Dice fall from 122 to 50.
Three downstream uses. Cobb angle on AASCE: measurable curves on all 218 films, MAE 5.13°, under the 6–7° inter-observer range; 2.08°, 4.12°, and 7.50° for mild, moderate, and severe curves; 61% within 5°, 88% within 10°. 2D/3D registration: a bidirectional Chamfer term on FleXray masks inside xvr. From a manual frontal init (median 324.5 mm, 100% failure) median mTRE drops from 335.3 mm to 1.2 mm and the failure rate from 86.4% to 12.7%. From a pose foundation-model init, all eight failures worse than 50 mm fall below 10 mm; worst case is 21 mm. Few-shot transfer: on pediatric hip dysplasia (MTDDH), 6 patients (1% of training) fine-tune to mean Dice 0.896 versus 0.867 for nnU-Net from scratch. On bone-tumor detection (BTXRD), 10% of data (130 cases) yields 23% AP50 versus 7%.
The bottleneck for quantitative X-ray is the training distribution, not a new architecture. Transfer depends on covering anatomy, geometry, and appearance; photorealism of any single DRR is secondary. A from-scratch U-Net and a finetuned SAM2 Large are statistically tied on the same engine (0.870 vs 0.876, interval includes zero). The recipe is worth more than a natural-image prior.
Weights, code, the enhanced DRR set, and a local browser tool are public. Groups with X-ray archives who want batch morphometry, or a semantic term in 2D/3D registration, can try it now. This is a solid engineering transfer of whole-body CT/MRI segmentation to radiographs via simulation, not a new network family.
The authors are direct about this. Few datasets have ground truth, and none of the quantitative tests include films with heavy clinical text overlays, which are common in practice. CT sources are mostly supine adults with extremities out of view. 764 real limb X-rays only partly close that hole, and unseen geometries still fail. Pathologies, implants, and hardware are not synthesized beyond what already sits in the source CTs, so severe trauma breaks the model.
A few extra caveats. The 18 Radiopaedia in-the-wild cases are qualitative only. Ensembling plus TTA lifts the single model from 0.866 to 0.875, a small gain, at 1.24 s latency and 2.36 GB peak GPU memory. BTXRD's test set is all tumor-positive, so AP50 is not screening performance. The Cobb pipeline needs at least three usable vertebrae; severe-curve error of 7.50° already sits at the top of inter-observer noise.