Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation
Jiazhen Liu, Mingkuan Feng, Long Chen
cs.CV
2026-08-04
STAMPlus splits MLLM segmentation into autoregressive dialogue and one-pass masks; an ID list drives a multi-class head, and one checkpoint covers five segmentation settings at lower latency.
Multimodal large language models (MLLMs) aim to fold vision tasks into a single instruction-driven framework. That has largely worked for recognition and detection, but segmentation is stuck on a trilemma: be accurate, preserve the model's dialogue ability, and stay fast at inference.
The root cause is that the MLLM's autoregressive text generation is ill-suited to emitting dense pixel masks. The two established routes each compromise. Embedding prediction (LISA, GSVA, PixelLM, READ) has the MLLM output a special token's embedding to drive an external SAM-style decoder; it is efficient for one target, but pixel-level supervision interferes with language modeling and dialogue suffers, so LISA may answer "how many objects?" with a segmentation. Next-token prediction (VisionLLM coordinates, Seg-Zero CoT coordinates, Text4Seg patch labels) keeps the token interface and needs no external decoder, but it autoregressively generates a long mask-token sequence per target, making high-quality dense masks prohibitively slow. Both are fundamentally one binary target at a time, so multi-target settings (semantic, instance) force repeated runs or ambiguous masks. STAMPlus tackles this, as an extension of the same group's STAMP (CVPR 2026).
The core idea is to decouple dialogue from mask generation, which the authors call All-Mask Prediction.
In Phase 1, the MLLM autoregressively generates its reply as usual and ends by emitting an in-vocabulary <SEG> token to trigger segmentation. Unlike LISA, this token is a standard vocabulary item, not an embedding tied to an external decoder, so no pixel-level supervision is introduced and dialogue ability is preserved. The KV states are cached.
In Phase 2, image-aligned [MASK] placeholders, one per patch, are prefilled into the model. Each placeholder's initial embedding is fused with its patch feature and positional encoding into a visually-augmented mask embedding. A single non-autoregressive forward pass with hybrid attention (causal over the dialogue history, bidirectional among mask tokens) classifies all patches as foreground or background at once. Optionally, a frozen SAM refines the patch-level result into a high-resolution mask.
STAMP is the binary instance of this paradigm, emitting one foreground or background mask. STAMPlus extends it to multi-target with two structural changes:
Because each patch pairs with one mask token, spatial granularity tracks the input resolution. Natural images use 1024 to 1280 mask tokens; remote-sensing small-target scenes raise both the input resolution and the token budget to 2560 to 3200, preserving fine evidence for small targets while leaving the structured mechanism unchanged.
Training uses one unified objective: text generation loss plus mask prediction loss. STAMPlus is trained on a single mixture (RefCOCO family, gRefCOCO, ReasonSeg, remote-sensing RRSIS-D and EarthReason, COCO-Stuff, COCO Panoptic and MUSE, LLaVA-665k) with one checkpoint shared across all tasks and no task-specific fine-tuning. It is built on Qwen2-VL in 2B and 7B sizes.
On single-target referring and reasoning segmentation, STAMPlus holds STAMP's ground and edges it up. On the RefCOCO family average cIoU, STAMPlus-7B scores 81.0 and STAMP-7B 80.7, both above the prior best Text4Seg++ (78.9). On ReasonSeg, STAMPlus-7B reaches 64.0, ahead of READ's 61.1. On remote sensing, RRSIS-D averages 76.2 (strongest specialized baseline 74.4, Text4Seg++ 70.8) and EarthReason 74.0 (Text4Seg++ 70.1). High-resolution scaling adds 5.5 gIoU on RRSIS-D and 4.9 on EarthReason in controlled comparisons.
The new multi-target abilities:
| Task | Metric | STAMPlus-7B | Comparison |
| Open-vocab semantic (ADE20K/PC/PAS avg) | mIoU | 63.7 | best |
| Instance-aware (MUSE, test) | gIoU/cIoU | 65.8/66.9 | Text4Seg++ 63.2/63.8 |
This is beyond STAMP: to separate instances, STAMP must repeat binary prediction with an unambiguous referring expression per instance, while STAMPlus discovers and distinguishes multiple instances from one instruction in a single structured pass.
Efficiency is the headline. In the single-target setting, STAMP's latency matches embedding-prediction methods and is far below next-token methods. In multi-target settings, STAMP and LISA repeat once per category and latency climbs steeply, whereas STAMPlus's added cost sits almost entirely in Phase 1 list generation, with all categories sharing one Phase 2 pass. Twelve-category latency drops from 13.50 s for repeated STAMP inference to 5.16 s.
Two diagnostics stand out. Supplying human-annotated target boxes in Phase 1 lifts STAMPlus-2B's average referring cIoU from 77.4 to 89.1, showing that Phase 1 description quality directly drives Phase 2 segmentation and pointing at an improvement path (reinforcement learning or external localization tools for Phase 1). In the other direction, the spatial grounding from segmentation training feeds back into understanding: under a look-twice protocol, STAMPlus-2B averages 71.1 on TextVQA, InfoVQA, POPE, and DocVQA, 2.6 above the same-initialized Qwen2-VL-2B, better on each benchmark.
For anyone building MLLM dense prediction, the interface of autoregressive dialogue plus one-pass non-autoregressive masks is a transferable paradigm: it sidesteps both the dialogue damage from pixel-level supervision and the slowness of next-token mask generation, and the structured target list lets one model naturally support multi-category and multi-instance segmentation. One checkpoint covering five segmentation families at competitive latency is a solid engineering result.
The ablations pin down the parts: removing hybrid attention or the visually-augmented mask embedding clearly hurts accuracy; resolution is adjustable for a latency-accuracy trade-off; and an LLaVA (Vicuna-7B) backbone variant still averages 77.0 cIoU, so the paradigm is not tied to Qwen2-VL.
As a journal extension, the paper states few limits of its own; a few deserve flagging. The multi-class head has a hard cap of 200 classes; no evaluated sample hits it, but panoptic segmentation of urban scenes with hundreds of instances would press that ceiling. The remote-sensing high-resolution scaling (2560 to 3200 tokens) buys accuracy with extra compute, and the paper does not give a full latency accounting for it.
The efficiency comparison deserves a discount: STAMP and LISA were not designed for multi-target, so using their repeated runs as the baseline is a favorable framing. The segmentation-feeds-understanding result is preliminary; the authors note they use an attention proxy rather than feeding discrete masks back into VQA (STAMPlus is not trained for that), so it is evidence that segmentation training sharpens internal spatial attention, not a closed loop. And because the whole paper is one group extending its own STAMP, the trilemma-resolved claim is strong; the benchmarks are the standard set, and cross-setting robustness still needs community validation.