Extending TotalSegmentator: Predicting Patient and Acquisition Characteristics from CT and MR Images
Jakob Wasserthal, Joshy Cyriac, Michael Bach, Kimia Mozahheb Yousefi, Minh-Son To, Máté Sik, Cédric Hémon, Thomas Weikert, Martin Segeroth
cs.AI
2026-08-30
A 3D ResNet-10 ensemble predicts weight, height, age, sex, and scan metadata from CT and MR. Internal CT MAEs are 3.90 kg, 3.68 cm, and 4.42 years; CPU inference takes 20 seconds.
PACS archives sit on hundreds of thousands of CT and MR exams, while the pipelines that consume them still trust DICOM headers. Weight, height, age, and sex change contrast dose, chemo dosing, and risk scores. Tube voltage, reconstruction kernel, and noise decide whether a radiomic number is comparable across scanners. Contrast timing is the difference between arterial and portal phase; the MR sequence label decides which analysis code is even allowed to run. Those fields are often blank, stale, or guessed at the bedside.
Earlier papers usually attacked one target: weight from a scout, weight from dose reports, height from tissue volumes, or a dedicated classifier for CT contrast or MR sequence type. What was missing was an open-source model that, in one forward pass on heterogeneous CT and MR volumes, recovers both patient traits and a long list of acquisition properties.
University Hospital Basel pulled clinical exams from January 2011 to August 2025: 57,291 CT studies (34,257 patients) and 43,200 MR studies (29,073 patients). Training required valid DICOM weight (35–160 kg), height (125–220 cm), age (12–100 years), and binary sex. Torso exams were preferred; head, joint, extremity, and low-information protocols were deprioritized. Internal tests held 501 CT and 636 MR exams with mixed fields of view. External testing used 54 Spine-Mets-CT-SEG CTs from TCIA, also cropped to thorax-only and abdomen-pelvis to measure how coverage moves error.
Each modality gets a 3D ResNet-10, five-fold ensemble. Volumes are reoriented, resampled to 2 mm isotropic, then center-padded or cropped to 240³ for CT and 210×210×150 for MR. One shared backbone, one linear head. Continuous and encoded categorical targets are z-scored together and trained with Huber loss (β=1) for 40 epochs, Adam at 5×10⁻⁴, random crops and per-axis flips.
Both models predict weight, height, age, sex, contrast presence, cranial and caudal vertebral coverage, and a noise score. CT also predicts manufacturer, tube voltage, tube current, a vendor-harmonized kernel sharpness code, and post-injection time. MR also predicts sequence class (T1, proton density, T2, FLAIR, STIR, T2, susceptibility-weighted, diffusion-weighted, angiography, or other). Most labels come from DICOM; vertebral bounds are the first and last C1–L5 vertebrae with TotalSegmentator volume above 100 voxels.
The baseline is not a toy classifier. It runs the full TotalSegmentator stack, then feeds organ, bone, fat, and muscle volumes plus median intensities into target-specific XGBoost models. That baseline needs a segmentation pass; the CNN reads the volume directly.
On the internal tests the CNN beat XGBoost on every core target. All eight paired comparisons stayed significant after Holm correction (CT adjusted P≤.042, MR ≤.030).
| Modality | Method | Weight MAE | Height MAE | Age MAE | Sex F1 |
| CT (n=501) | CNN | 3.90 kg | 3.68 cm | 4.42 yr | 0.990 |
| CT | XGBoost | 4.90 kg | 4.29 cm | 6.60 yr | 0.969 |
| MR (n=636) | CNN | 4.34 kg | 4.62 cm | 7.13 yr | 0.970 |
| MR | XGBoost | 7.08 kg | 5.21 cm | 11.46 yr | 0.932 |
Extra heads: CT contrast F1 0.963, manufacturer micro-F1 0.988, post-injection time MAE 5.36 s, cranial/caudal vertebral error 0.75 / 0.21 levels. MR sequence micro-F1 0.953, contrast F1 only 0.823, vertebral bounds 2.83 / 2.50 levels. Tube voltage MAE 5.58 kV is usable; tube current at 165.84 mA is not. Noise scores: 1.54 (CT), 2.34 (MR).
Joint training of the four core targets matched four separate models (adjusted P≥.589). Adding the extra CT heads left weight, height, and sex unchanged, but age MAE rose from 4.02 to 4.42 years (P<.001). Multitask is cheap, not free.
On 54 external full thorax-abdomen-pelvis CTs: 4.45 kg, 4.05 cm, 5.17 years, sex F1 0.971. The CNN–XGBoost gap was no longer significant after correction. Abdomen-pelvis crops favored the CNN on weight (3.78 vs 7.63 kg). Thorax-only age improved (5.86 vs 11.53 years) while weight error rose to 6.24 kg. Coverage decides whether a body-stat prediction is even plausible, which is why the paper wants predicted vertebral bounds returned with the estimate.
Five-fold CPU inference took 20 s and 3.8 GB on a 512×512×807 CT, 12 s and 1.7 GB on a 320×250×72 MR. Every modality-specific output comes out of the same forward passes.
This is a patch for archives and routing, not a replacement for a scale in clinic. When DICOM is empty or wrong, the model can flag implausible metadata, clean retrospective cohorts, audit protocols, and send an exam to the right downstream code based on sequence and coverage. BMI and BSA can be derived with the Mosteller formula, but high-risk dosing and ventilator tidal volume still need a tape measure.
The models are open source, run on CPU, and already live in TotalSegmentator, so groups that already segment with it get a near-zero integration cost. Age error of 4–7 years is fine as a consistency check or an imaging biomarker. It is not a forensic age.
Most ground truth is DICOM or another algorithm, so labels can be estimated, stale, or wrong. Kernel codes and noise scores are technical surrogates without a clinical unit. Patients under 18 are underrepresented. External validation is 54 CT exams, with no independent MR set and no multicenter test of the extra heads.
Training itself required complete DICOM body stats, so the network learned to invert fields that were already present. Whether that holds on exams whose headers are actually missing is untested.
Tube current MAE of 165 mA is a placeholder output. MR contrast F1 of 0.823 and vertebral errors of about 2.5 levels lag CT; the paper points to sequence heterogeneity and missing vertebral labels. The authors say these predictions should not replace verified clinical measurements.