WeVisOne-VL: Unified Visual Perception from Objects to Pixels
Tianming Liang, Junyi Pan, Yuxi Li, Canmiao Fu, Zhipeng Huang, Chong Sun, Chen Li, Jing Lyu
cs.CV
2026-10-22
WeVisOne-VL (2B/4B on Qwen3-VL, plus a 9.6M decoder) unifies detection, segmentation, and matting. The 4B scores 71.3 mean F1 on Dense200 and 67.4% absent-object rejection.
A general vision-language model can say what is in an image and still fall apart on where it is, how many there are, or where the boundary falls. Objects that are absent get a box. Instances that are not salient get missed. Detectors and the SAM family are steadier at localization, but boxes, points, masks, and continuous opacity each live in their own model. A different output granularity means chaining on another system.
WeChat Vision at Tencent builds WeVisOne-VL on Qwen3-VL, at 2B and 4B. One natural-language request can return points, boxes, a binary mask, or an alpha matte. The tasks have to supervise the same instance. When the target is truly absent, the model has to return nothing.
The data engine covers about 100 million images and 3.6 billion boxes, drawn from GRIT, CC3M, LAION, and SA-1B. Categories and referring expressions sit in one knowledge graph, which also decides what to collect next. Detection boxes come from SAM3 together with an in-house DINO-style detector; Qwen3-VL checks the category and removes duplicates. Referring boxes come from the SAM3 Agent and Rex-Omni, then the same check covers attributes, states, actions, and relations. SAM2 adds a visible-region mask that inherits the original query, so the box and the mask name one instance.
A negative requires that no object in the image satisfies every condition. An empty detector output is not evidence, because the detector may have missed the target. The sample becomes a hard negative only when the current model still draws a box, and the label is an empty set. Labels change only after humans review the errors. Disagreement alone does not rewrite them.
For matting, BiRefNet and MatAnyone2 predict alpha on the coarse mask. Detail is kept when the two agree spatially, and the pipeline otherwise falls back to the binary mask. A refinement head on a frozen SAM2.1-Base+ then re-labels the corpus with continuous opacity.
Every task is a structured next-token string. Coordinates use dedicated tokens from <0> to <1000>, four per box. An absent target is None. Pixel tasks append <OBJ> after the box, and that hidden state represents the instance. A decoder of about 9.6M parameters reads multi-level ViT features, this state, and the predicted box. Six deformable-attention layers and a dynamic kernel produce a full-image mask, and three upsampling steps repair thin boundaries. Inference does not call SAM.
Supervised training has three stages. Stage 1 uses about 90.2M samples for 13K steps and updates only the language model, with at most 50 boxes per category, so dense pages do not collapse into repetition. Stage 2 uses about 41.8M samples for 9K steps, unfreezes the vision encoder, raises the cap to 1,960 visual tokens, and allows 500 boxes. Stage 3 warms the decoder for 100K steps on 250M segmentation samples, with 20 times focal loss plus Dice. Only the decoder and the <OBJ> embedding are kept and attached back to the stage-2 checkpoint, then token loss and pixel loss alternate one to one. The language model is frozen, and the decoder trains for another 100K steps on 130M matting samples, 99.8% of them from this engine. GRPO then selects about 140K uncertain prompts with the score 4μ(1-μ)σ, filled out to 178K. Each prompt draws 8 full outputs. The reward cubes the IoU of matched boxes and divides by the sum of predicted and ground-truth counts, so misses and duplicates both lose. A negative scores only when the model predicts nothing. For segmentation and matting, reinforcement learning still scores boxes only.
Decoding is greedy, at temperature 0, with at most 1,960 visual tokens. These models have no calibrated confidence, so detection is F1, not AP. COCO and LVIS are zero-shot. Qwen3.5 is run with thinking mode, which scored higher in this evaluation. The table is mean F1 over IoU from 0.5 to 0.95.
| Benchmark | 4B | 2B | VLM baseline | GroundingDINO-T |
| COCO | 57.1 | 55.2 | LocateAnything-3B 54.7 | 56.6 |
| LVIS | 55.3 | 53.3 | LocateAnything-3B 50.7 | 38.8 |
| Dense200 | 71.3 | 61.4 | LocateAnything-3B 58.7 | 33.1 |
| VisDrone | 44.7 | 40.4 | LocateAnything-3B 39.9 | 38.5 |
On COCO the 4B model is 0.5 above GroundingDINO-T. At IoU 0.95 it scores 19.5, and GroundingDINO-T scores 23.0. Crowded scenes are where the gap opens: on Dense200, Qwen3.5-4B scores 18.9, against 71.3.
Mean F1 on RefCOCOg test is 81.8 for the 4B model, 78.2 for the 2B, 77.6 for LocateAnything-3B, and 74.9 for Qwen3-VL-8B. Absent-target rejection on RefCOCOg-Rej is 67.4% for the 4B, 50.2% for the 2B, and 22.5% for Qwen3-VL-8B. Rex-Omni and LocateAnything stay under 1%. The in-house WeRefBench has 2,880 Chinese and English queries. The 4B model scores 78.7 mean F1 and 96.4% rejection, against 51.7 and 73.0% for LocateAnything.
Counting is exact match. Off by one is wrong. On PixMo-Count val/test the 4B model scores 93.7%/89.6%, and LocateAnything scores 88.7%/82.8%. On MixCount the 4B scores 42.0%/40.4%, against 12.5%/15.2%. On CountQA it scores 57.1, LocateAnything 52.4, and the 2B model 47.9.
PointBench is 71.9 for the 4B model and 69.9 for Rex-Omni-3B. PixMo-Points is 74.7, 73.2, and 72.9 for the 4B, the 2B, and Rex-Omni.
OCR requires an exact transcription and a box that clears the IoU threshold. Mean F1 for the 4B model is 39.0 on HierText, 47.8 on ICDAR2015, 52.0 on TotalText, and 68.1 on SROIE, against 30.5, 25.6, 25.7, and 58.6 for PP-OCRv5. At IoU 0.95 every system is still in single digits.
Referring-segmentation cIoU on RefCOCO is 81.4/84.5/80.1 for the 4B model and 80.8/83.0/77.4 for UniPixel-7B. Most of the margin is under two points. On ReasonSeg, without chain-of-thought, the 4B gIoU is 71.2/66.8, and LISA-13B fine-tuned is 65.0/61.3 in the same table. On Ref-ZOM the 4B rejection rate is 86.0% at 72.2 cIoU, GSVA-7B rejects 41.9%, and UniPixel rejects 0.3%. Mask overlap moves by a small margin. Refusal is where the scores separate.
Matting is lower-is-better, and MAD is scaled by 1,000. On P3M-500-NP and PPM-100 every method receives the ground-truth box, so the test is the boundary from the same starting box. MAD on P3M is 6.6 for the 2B model and 7.3 for the 4B, against 15.4 for SAM2-Matting with SAM3. SAD on PPM-100 is 49.3, 57.2, and 92.9 for the 2B, the 4B, and SAM2.1-Base+. Language-conditioned matting is the in-house WeRefMat set: SAD is 39.0 for the 4B model, 82.3 for Qwen3.5-9B plus SAM2-Matting, and 162.2 for SAM3 text grounding. On P3M and PPM-100 the 2B model beats the 4B.
Detection, counting, OCR, referring segmentation, and language-conditioned matting share one interface. The pixel decoder is about 9.6M parameters, SAM is not required at inference, and the code and weights are public.
Against Qwen3-VL, Qwen3.5, Rex-Omni, and LocateAnything, the gap sits in dense detection, counting, OCR, and rejection. Against GroundingDINO, COCO mean F1 is only 0.5 higher. RefCOCOg-Rej moving from under 1% to 67.4% shows that an empty target set can be learned in the same structured generation.
The paper states four limits. Perception training may hurt general ability relative to the base model, and no MMMU-style number is reported. The model emits structure directly, with no chain-of-thought, so hard reasoning will be weaker. It is single-turn, and multi-turn tool use is left for later. Benchmark prompts are clean and unambiguous, while real requests are vague, and intent sits outside this evaluation.
Reinforcement learning for segmentation and matting never scores the mask or the alpha, so pixel quality comes from supervised training. Most labels come from SAM3, SAM2, Qwen3-VL, Rex-Omni, and matting experts. Human review is a spot check, and no ablation removes one expert. Some of the sharpest rejection and language-matting numbers sit on the in-house WeRefBench and WeRefMat. Public sets agree in direction. Coordinates use 1,000 bins, and COCO F1 at IoU 0.95 is still 19.5. Latency and memory are not reported.