Xiaomi-Lens-0 Technical Report
Pei Fu, Yike Liu, Yue Sun, Hanmo Wang, Xiuwen Xi, Ruoceng Zhang, Shaojie Zhang, Chao Jiang, Hang Li, Bingrui Liu, Zhenbo Luo, Heng Qu, Changjian Xu, Ming Xu, Jian Luan
cs.CV
2026-10-16
A 4B Qwen3.5-based MLLM decodes masks from its own visual features so mask supervision reaches the shared backbone: +5.6 average REC across eight RefCOCO-family splits.
Multimodal LLMs answer whole-image questions well but fail on region-level detail: they miscount adjacent objects and misread attributes of small areas. The Xiaomi team traces this to text-only supervision. The training signal comes entirely from the text answer; nothing in the loss says which pixels support the answer or where the region ends, and a bounding box encodes only a coarse rectangle.
Segmentation-capable MLLMs already train with mask supervision, but most get their spatial detail from add-on components: LISA and Sa2VA attach a hierarchical segmentation encoder, Qwen3-VL-Seg fuses multi-scale features from a supplementary encoder. PixelLM and MLLMSeg decode from the MLLM's own vision encoder but keep it frozen, so the mask loss never reaches the shared features. The question this report asks: if mask supervision is pushed directly onto the MLLM's own representations, does the benefit spill over from segmentation into fine-grained perception and reasoning?
Xiaomi-Lens-0 is a 4B model built on Qwen3.5 with one small decoder head attached. The key choices:
"Thinking with visual latents" follows directly from this design: a chain of thought inserts a <|mask|> token wherever it references a region, and that latent is decoded into a supervised mask. A counting trace can assign one latent per instance. The data pipeline leans hard on verification: referring data fix the target before synthesizing instructions, then re-ground each instruction to confirm it selects that target alone; reasoning data have an MLLM write questions, an independent model answer them blind, and a reasoning model generate traces whose box references are paired with instance masks or SAM 3 masks, kept only when box IoU passes a threshold. At inference, masks are an output branch; nothing is fed back to the language model. Training runs in two stages, decoder-only pretraining on a frozen backbone, then joint fine-tuning of everything.
On REC, the eight splits of RefCOCO/RefCOCO+/RefCOCOg all beat the backbone, by 5.6 points on average and 7.6 on RefCOCO+. Against baselines it ranks first on seven splits and second on RefCOCOg test by 0.5. The margin widens at stricter thresholds: at IoU 0.95 the gain over the backbone reaches 15.5-16.8 points, so the boxes are precisely placed, not just pointed at the right object.
| Task | Metric | Xiaomi-Lens-0 | Comparison |
| RES, 8 splits | avg cIoU | 81.4, first on all | Qwen3-VL-Seg 78.6; SAM 3 beaten by 7.0-12.9 |
| COCO 2017 multi-instance | F1@mIoU | 63.9, first | GroundingPI 63.0, GroundingDINO 60.6 |
| ReasonSeg test | gIoU/cIoU | 68.0/62.5 | beats InstructSAM by 6.9/1.5 |
| RoboRefIt gIoU | 87.0 TestA, 81.5 TestB | beats best baseline by 4.5/7.1 | InstructSAM |
Text-only perception benchmarks also beat the backbone across the board:
| Benchmark | Qwen3.5 | Xiaomi-Lens-0 | Gain |
| MME-RW-cn | 60.3 | 67.5 | +7.2 |
| CountQA | 43.8 | 50.6 | +6.8 |
| V | 82.2 | 87.4 | +5.2 |
| HR-8K | 75.6 | 80.6 | +5.0 |
| CV-Bench | 85.4 | 88.1 | +2.7 |
Ablations separate the contributions. A BBox-only variant already gains 3.1-4.9 REC over the backbone; adding mask supervision adds another 1.1-2.4 (largest on RefCOCO+) and lifts all six perception metrics. Grounding data drive most of the referring gain (grounding-only reaches 94.5 on RefCOCO val); grounded reasoning traces add only 0.1-0.6 REC and 0.5-0.8 RES, concentrated in mask quality.
The self-built LensBench uses 1,200 paired units (two ground-truth masks per image × four instructions crossing explicit/implicit with single/multiple targets, across general, aerial, driving, and panoramic imagery, every unit human-reviewed). On explicit instructions it leads with 71.4 cIoU; on implicit ones it falls to 54.7, fifth behind Qwen3.8-Max (64.5), GPT-5.6-Sol, Gemini-3.6-Flash, and Seed-2.1-Pro. Its biggest domain win is aerial (57.1 cIoU, 7.0 over the best baseline); on general imagery it trails Qwen3.8-Max 63.9 to 74.2.
The report closes a causal loop: mask gradients flow through shared patch features and latent states back into the backbone, and the BBox-only ablation shows those gains are real and spill into text-only benchmarks like CountQA and V. For teams building grounding, "no segmentation encoder, mask supervision straight into shared representations" is a reusable recipe, and at 4B it outperforms encoder-equipped 7B/8B systems such as LISA and Text4Seg on RES. LensBench's paired design (same image and ground truth, only wording and target count vary) separates "picked the wrong object" from "drew the mask poorly," an evaluation idea worth copying on its own.
Implicit instructions are the weak spot: cIoU drops 16.7 points from explicit to implicit, and on panoramic multi-target cases [email protected] falls from 81.3 to 48.7; the authors flag this as an open problem. The "think with visual latents" framing runs ahead of the numbers: decoded masks are never fed back, so at inference the model cannot see the regions it decoded. The mechanism provides training supervision and inspectable outputs, not o3-style image interaction, and the traces' measured gains are small. On COCO multi-instance grounding, F1 at IoU 0.50 (80.4) still trails GroundingDINO and GroundingPI; the lead is concentrated at strict thresholds. The data pipeline relies on models checking models (SAM 3, MLLMs, a reasoning model), and the paper does not quantify whether their errors systematically bias the training set. Main results average three checkpoints with no variance reported, and LensBench comparisons were all run by the authors.