ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Lirong Wu, Sicheng Yang, Jinwang Wang, Pengju Wang, Zhitao Zeng, Yizeng Han, Yan Xing, Shengxuan Luo, Tao Feng, Qing Xie, Weigen Yao, Yi Yang, Zuozhu Liu, Jiasheng Tang, Shaocheng Wang, Jitao Wang, Jiahong Dong, Weihua Chen, Feng Xu, Fan Wang
cs.CV, cs.AI, cs.CL
2026-07-28
ClinFusion unifies 2D and native 3D medical imaging with a cascaded encoder and RoI-grounded evaluation, taking 20 of 24 medical benchmarks and beating GPT-5.2/Gemini-3-Flash on 13 of 16 multimodal benchmarks.
Deploying multimodal LLMs in medicine is, as the authors frame it, a vision-centric challenge: models must absorb heterogeneous 2D (X-ray, pathology, fundus) and 3D (CT, MRI) images, and evaluation must align with how radiologists actually work. Existing medical MLLMs struggle on both fronts, either slicing 3D volumes into 2D and losing structural information or requiring laborious 3D alignment, and they evaluate with surface lexical matching (BLEU, ROUGE) or coarse LLM scoring that is neither fine-grained nor factuality-driven.
ClinFusion builds on Qwen-VL with a compositional cascaded vision encoder and a Cascade Spatial-Aware Locality (CaSL) Fusion operator.
The encoder ensemble has a foundational Qwen ViT plus two 2D specialists (ConvNeXt for local high-frequency texture, DINOv2 for self-supervised semantics) and a native 3D encoder pre-aligned with language via contrastive learning on medical volumes. CaSL Fusion uses local cross-attention to fold these features into the Qwen ViT representation in stages: for each token it takes only a k-by-k neighborhood from the other feature map, attends, and adds the residual; multiple encoders cascade left-associatively with the base representation always as query. Training adds stochastic residual regularization (Bernoulli drop 0.1). 3D volumes take a dual path: native 3D encoding plus 4 representative slices as 2D anchors, with depth-aware 3D CaSL letting anchor features attend to volumetric context across depth.
The evaluation framework is rebuilt too. MedIF-Bench (900 samples, 7 task categories) tests instruction following by checking format compliance via regex, not medical correctness. Report generation uses a three-stage RoI-grounded protocol: extract clinical indication and areas of focus from the ground-truth report (strictly guarded against leakage), generate with that context, then an LLM judge splits diagnostic claims into matched, missed, and hallucinated to compute Precision, Recall, and F1, which is more tolerant of paraphrase and less biased to length than RadGraph-F1.
Training is progressive across four stages: general vision-language alignment, medical knowledge injection, instruction fine-tuning, and 3D volumetric adaptation.
ClinFusion takes 20 of 24 benchmarks against medical MLLMs and beats GPT-5.2 and Gemini-3-Flash on 13 of 16 multimodal benchmarks.
| Benchmark | ClinFusion | Comparison |
| AMOS-MCQ (3D VQA) | 8B 80.2 | Hulu-Med-32B 73.9; Gemini-3-Flash 16 points lower |
| CT-Rate Report F1 | 32B 23.9 | Gemini-3-Flash 20.2; GPT-5.2 14.1 |
| MedIF-Bench instruction following | 8B 98.1 / 32B 98.9 | GPT-5.2 96.0; Gemini-3-Flash 96.6 |
| CheXpert-Plus Report F1 | 37.8 | Hulu-Med-7B 31.9 |
Six board-certified radiologists (6+ years) blindly evaluated 300 cases: ClinFusion with agentic tools ranked first on accuracy, completeness, and operability, significantly ahead of Hulu-Med and Gemini-3-Flash (p<0.001). The RoI metric had the highest correlation with expert judgment among 11 automatic metrics (Kendall tau=0.511).
Medical imaging is one of the few verticals where MLLMs have real deployment value, but 3D imaging (CT, MRI) has long been a weak spot for open models. ClinFusion unifies 2D and native 3D in one encoder and aligns evaluation with how radiologists actually judge, pushing two engineering problems at once. Both model and evaluation protocol are open, usable as a base and a benchmark for medical AI teams. It also demonstrates the value of designing evaluation around the clinical workflow itself.
The agentic toolset is still small; the authors plan to extend it to ultrasound, MRI, and treatment planning. A gap remains on knowledge-intensive text benchmarks versus proprietary models, attributed to capacity and training-data scale. The paper itself stresses that real clinical translation needs rigorous human-in-the-loop diagnostic-accuracy and patient-outcome studies plus ethical and regulatory clearance, so the lead here is technical only. Model scale (8B/32B) and exact training data sizes are not fully disclosed.