P2Echo segments four echo views from text prompts, median Dice +2.33 at 6.90 GFLOPs

Prompt2Echo (P2Echo): Text-Prompted Multi-View Echocardiography Segmentation with One Model

Moein Heidari, Alvin Kimbowa, Mahdi Tabatabaei, Afshin Bozorgpour, Sina Ghorbani Kolahi, River Jiang, Wenjin Chen, David J. Foran, Jasmine Grewal, Dorit Merhof, Ilker Hacihaliloglu

cs.CV

2026-11-10

P2Echo segments four echo views with one structured text prompt. Internal Dice is 92.0 to 96.9, and the median gain over the strongest baseline is 2.33 points at 6.90 GFLOPs.

What problem this solves

Quantifying an echo means reading several standard windows, not a single frame. In apical two-, three-, and four-chamber views (2CH, 3CH, 4CH) the left ventricle is bullet-shaped. In the parasternal short axis (PSAX) it is a circle, and the chambers on screen change with the window. Scanner, site, and patient differences add a further shift, while speckle, shadowing, and dropout eat the boundary.

A specialist network per view dodges that geometry clash. It also multiplies training runs and the models a clinic has to keep, and the views do not stay equally accurate. Train a classic architecture on pooled planes without naming the window, and the fit slides toward an average that blurs both shapes. EchoONE covers several views with dense prompts taken from clustered structural prototypes. Those prompts are implicit, so an operator cannot name the view and the target. MedCLIP-SAMv2 and Grounding DINO-US-SAM use text to localize a structure or to mint a visual prompt. The words stop before the decoder and never retune features at every spatial scale.

Method

P2Echo steers one network with a fixed string, "View: v. Target: k. Description: δ(v, k)." The description is a short anatomical note for that view. Four-chamber LV is bullet-shaped and forms the apex. Short-axis LV is circular at mid-ventricle. Text is embedded by frozen Qwen3-Embedding-0.6B. The prompt list is closed, so the vectors are computed once and reused, and the language path is nearly free at inference.

The image backbone is ImageNet-pretrained PVTv2-B2, a pyramid vision Transformer with four feature scales. The deepest map and the prompt tokens pass through three cross-attention layers. The result is an image-conditioned mask embedding, projected to the channel width of each decoder stage.

Gluing text onto the end of the decoder does not fix a structure whose shape conflicts across windows. Alignment therefore sits inside every stage: shared redundancy is subtracted first, then attention follows the current prompt. That block is DyITA, followed by a gated 3x3 depthwise convolution for local edges. Routing uses a straight-through estimator. The forward pass picks one expert, and softmax gradients train the router.

The loss is boundary difference-over-union (BDoU). Region overlap stays in the objective, with extra weight where the contour disagrees. That weight tracks the ratio of boundary length to foreground area. The main head and two intermediate decoder outputs are supervised together. Inputs are 256x256, trained with SGD for 200 epochs, batch size 32, learning rate 0.01.

Five development sets use an 80/10/10 split. Classes are LV, myocardium, LA, RV, and RA. The only 3CH labels are LV masks from SegRWMA. The only PSAX labels are LV masks from EchoNet-Pediatric. CardiacNet, EchoNet-Dynamic, and HMCQU are held out. HMCQU annotates myocardium without an LV cavity, so the cavity is filled by joining the basal endpoints and closing the contour.

Results

Internal mean Dice is 96.94±1.72 on 2CH, 94.71±2.16 on 3CH, 93.87±5.07 on 4CH, and 92.00±4.97 on PSAX. HD95 is 2.03, 3.96, 3.33, and 6.99. Versus EchoONE the Dice gains are 5.3, 1.1, and 2.8 on the first three views, with parity on PSAX. CENet scores 93.11 and 93.09 on 2CH and 3CH. nnUNet scores 91.39 and 91.50 on 4CH and PSAX. Wilcoxon signed-rank tests against the runner-up give p<0.01 on the first three views. PSAX carries no significance mark.

On the qualitative panel, baselines more often drop or fuse myocardium and the right heart. In the illustrated 2CH case, EchoONE mixes the LV cavity with the myocardial wall.

SettingP2EchoReference
Internal 2CH/3CH/4CH/PSAX mDice96.94 / 94.71 / 93.87 / 92.00EchoONE +5.3 / +1.1 / +2.8 / parity
HMCQU mDice82.22nnUNet 45.30, ResUNet 58.02
EchoNet-Dynamic mDice87.79CENet 85.67; HD95 11.33 vs 11.89
CardiacNet mDice92.95CENet 89.56
Compute6.90 GFLOPs, 64.68 FPSEchoONE 174 GFLOPs; SAM3 2417 GFLOPs

Held-out scores are 82.22±9.53 Dice on HMCQU (HD95 6.92), 87.79±7.01 on EchoNet-Dynamic (HD95 11.89), and 92.95±10.02 on CardiacNet (HD95 3.37). The EchoNet-Dynamic boundary distance is not a significant win. On HMCQU, nnUNet falls to 45.30±45.41. Myocardium labels in training come almost only from CAMUS, and a standard segmenter can treat a missing class as anatomy that is absent. The reported median Dice gain versus the strongest baseline on each evaluated set is 2.33 points.

The model has 37.47M parameters. On an H100 in FP32 at batch 1 it takes 0.01546 seconds per frame, 1.51 times CENet and 4.28 to 159.36 times the promptable models in the comparison. SwinUNet is quicker and cheaper at 0.00577 seconds and 6.16 GFLOPs, and clearly less accurate. Zero-shot SAM3 barely transfers to these exams.

Removing dynamic projection, the measure kernel, or the difference operator on HMCQU costs 2.18, 1.70, and 2.19 mDice and adds 0.39, 0.43, and 0.92 HD95. The boundary gain is mostly the subtraction. A 4B text encoder instead of 0.6B moves mDice from 82.22 to 82.29 and HD95 from 6.92 to 6.95.

On 450 internal images and 950 structures, wording alone changes the score. The canonical template reaches 93.53±6.83 Dice, a clinical paraphrase 80.48±23.88, and a short command 35.83±36.23. An empty prompt and "segment the cardiac chamber" sit near zero. "Segment the liver" still scores 6.35.

Why it matters

A view and a structure written into the prompt let one network stand in for a stack of per-view models, and the control string is easier to audit than a clustered prototype. At 6.90 GFLOPs and 64.68 FPS the budget fits post-exam processing.

Against the convolutional numbers above, mDice is higher by 3.83, 1.62, and 2.48 on 2CH, 3CH, and 4CH, and by 0.50 on PSAX. That is an incremental segmentation gain. The wider gaps are HMCQU, where labels are incomplete, and the runtime against EchoONE and SAM-style adapters. Text functions as a switch. Moving from a 0.6B embedding model to a 4B one changes almost nothing, because those vectors can be cached ahead of time.

Limitations

Only four standard views are tested, and the prompt is built with the view identity already known. Unseen windows are untested. Three-chamber and short-axis supervision each come from a single development set, and both are almost entirely LV, so the right heart and the myocardium are thinly covered. The benchmark stops at masks. Volumes and ejection fraction are not measured, so it remains open whether cleaner contours produce more stable clinical numbers.

Rephrase the prompt and the score drops. The short command has a standard deviation of 36 Dice points, which rules out free clinical language for now. The HMCQU cavity is derived by closing a myocardial contour, so the large margin on that set has to be read with the label construction. A missed class keeps Dice 0 inside the average, while infinite HD95 values are discarded. Boundary scores therefore look slightly kinder to the baselines.

Terms

Source

What people are saying

Related papers

All paper explainers