LLMSeg: Training-Free Text-Prompted Image Segmentation with B-Splines
Abdullah Hamdi
cs.CV, cs.AI
2026-10-21
A frozen multimodal LLM writes B-spline mask geometry as text, no mask head, no training: 71.5 cIoU on ReasonSeg, beating SAM 3 on most of ten referring and reasoning splits.
Text-prompted segmentation has been specialist territory. SAM 3 gets open-vocabulary concept segmentation from a dedicated detector-tracker architecture plus a massive human-and-model-in-the-loop data engine. Multimodal LLMs have general visual skills like grounding, counting, and coarse localization, but the field's working assumption is that dense masks require a trained decoder bolted on, as in LISA or GLaMM. This paper asks the plain question: freeze an MLLM, attach nothing, train nothing. Can it segment? It can, and across a benchmark suite it often beats SAM 3.
The only vision component is an off-the-shelf frozen MLLM (Gemini-3.7-Flash in the main results). Three stages, each one structured JSON call:
The B-spline choice is deliberate. A handful of control points is short enough to serve as LLM output; each point influences only a few neighboring segments, so one bad point damages a bounded arc rather than the whole curve; and box confinement makes detection quality the ceiling of segmentation quality, so errors cannot spread across the image. Refinement amounts to the model inspecting its own rendered overlay and correcting its own boundary.
| Benchmark | Metric | LLMSeg | SAM 3 | SAM 3 Agent |
| ReasonSeg val | cIoU | 71.5 | 9.9 | 62.0 |
| ReasonSeg test | cIoU | 65.8 | 11.3 | 63.2 |
| RefCOCO testB | cIoU | 76.0 | – | 75.5 |
| LVIS | cgF1 | 53.9 | 50.1 | 51.0 |
| LVIS | mask AP | 47.2 | 62.2 | 50.9 |
| Cityscapes | mIoU | 53.3 | 63.7 | – |
| RF-100VL | AP0 | 31.4 | 14.2 | – |
SAM 3 Agent and LLMSeg share the same Gemini-3.7-Flash backend. SAM 3 with text-only prompts collapses on reasoning queries (9.9 cIoU on ReasonSeg val), because the target has to be inferred before it can be segmented. The 71.5 is the best result among methods not fine-tuned for the task; refinement adds 6.9 points (64.6 to 71.5), and the ablation without it loses 5.4 cIoU on ReasonSeg and 12.7 on RefCOCO.
The backend study swaps 21 models with the pipeline held fixed: stage-2 gIoU ranges from 13.4 to 74.2, and GPT-6 Astra hits 74.2 without refinement, above SAM 3 Agent's 72.4. Segmentation quality tracks backend upgrades with no retraining, though general benchmark scores predict it poorly (Claude Opus 5 is the counterexample). The cost is equally clear: 13.4 calls and 159 seconds per image on average, against a single forward pass for SAM 3.
This is the first demonstration at benchmark scale that dense segmentation can be delivered as text output from a general MLLM, with zero training and zero segmentation infrastructure. Anyone with an MLLM API can run reasoning segmentation, which is exactly where SAM 3 is weakest. Every stage emits human-readable JSON, so geometric decisions are inspectable. Long-tail domains are a bright spot: on RF-100VL its AP0 is 2.2 times SAM 3's. The fit is low-throughput, high-semantic-complexity work such as medical or rare-domain annotation; for dense scene labeling or real-time use, specialists remain the right tool.
The authors list four: detection errors, imprecise pixel-level boundaries, amorphous classes, and inference cost. The numbers back them up: Cityscapes mIoU trails SAM 3 by 10.4 points, and mask AP trails SAM 3 on both LVIS and COCO. The evaluation protocol has caveats: only four of ten benchmarks ran in full, the RefCOCO family used 500-item random samples per split (baselines rescored on matched items), and the LVIS/COCO AP columns rest on a few dozen surviving images, so they are not comparable to published numbers. The headline win over SAM 3 leans partly on SAM 3's text-only failure at reasoning queries; the fairer rival, SAM 3 Agent with an MLLM driver, stays stronger across the RefCOCO family. And 13.4 calls at 159 seconds per image is an order of magnitude slower than one forward pass of the 860M-parameter SAM 3.