Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
Lucy Lin, Ayush Jain, Yifan Liu, Katerina Fragkiadaki
cs.CV
2026-08-04
Qwen-3D (CMU) makes a VLM attend directly in 3D world space, matching specialist models on grounding (within 0.7 pts on ScanRefer) and doubling prior generalists on instance segmentation, with 2D skill intact.
Vision-language models handle single images well but struggle with long, multi-view video streams of a room or scene. The reason is blunt: attention scales quadratically with sequence length, and context windows are finite, so stacking tokens frame by frame blows up fast.
3D geometry offers a natural compression lever. With depth and camera pose for each frame, every view can be projected into a shared world coordinate frame and merged into one persistent, scene-aligned representation. Two frames far apart in time that observe the same corner end up adjacent in 3D.
A line of 3D LMMs has followed this route, but they still trail specialist 3D perception models on grounding and segmentation. The authors argue the bottleneck sits in decoding: existing methods push 3D predictions out through language tokens, proposal selection, or a lightweight <REF> query token, inserting a layer between language reasoning and dense geometric prediction.
Qwen-3D builds on Qwen2.5-VL and threads geometry through representation, attention, and decoding.
First, a geometry-aware scene representation. RGB-D frames with depth and pose are projected into a shared 3D frame and merged with 5cm voxel pooling: features and coordinates inside each voxel are averaged. Hundreds of multi-view frames collapse into a set of unordered 3D tokens. This step is load-bearing; without voxel pooling the model runs out of memory even at much lower frame counts.
Second, geometry-aware attention. Qwen2.5-VL uses 2D multimodal rotary position embeddings (mRoPE), splitting position into time, height, and width. Qwen-3D swaps in 3D rotary embeddings, splitting PE3D into time, x, y, and z, so attention operates in world space rather than within independent image frames. It also replaces causal masking over visual tokens with full attention: a non-autoregressive task like grounding has no reason to let each token see only its predecessors.
Third, and most important, geometry-aware decoding. A Mask2Former-style query decoder routes learnable object queries through alternating cross-attention to visual and language tokens, with each query predicting both a segmentation mask and a text span. Language is grounded directly in the 3D scene representation instead of squeezed through <REF> tokens. This unifies referential grounding, instance segmentation, and question answering over both images and video.
Training is joint 2D and 3D: roughly 255k 3D samples (ScanQA, SQA3D, ScanRefer, ScanNet) plus the RefCOCO family and LLaVA-Instruct on the 2D side. 2D data is lifted to 3D with some probability using reconstruction models such as MoGE. Only about 50M new parameters are trainable, and the backbone is fine-tuned alongside.
Table 1, 3D grounding and QA (ScanRefer uses [email protected], both QA benchmarks use EM@1):
| Method | [email protected] | ScanQA EM@1 | SQA3D EM@1 |
| UniVLG (expert, SOTA) | 63.5 | 25.7 | 50.2 |
| LLaVA-3D-7B (prior LMM SOTA) | 50.1 | 27.0 | 55.6 |
| GPT-4o (2D VLM) | 48.2 | 18.0 | — |
| Qwen-3D-7B | 62.8 | 31.5 | 59.6 |
| Qwen-3D-3B | 62.2 | 28.0 | 55.4 |
On ScanRefer, Qwen-3D-7B's 62.8 sits just 0.7 points behind the specialist UniVLG at 63.5, and on both QA benchmarks it leads everyone: ScanQA EM@1 31.5 and SQA3D 59.6. It beats the prior LMM state of the art LLaVA-3D by 12.7 points and GPT-4o by 14.6. Most of these numbers come from a 3B backbone that matches or beats methods running 7B or larger.
Out-of-domain generalization is even starker. On ScanNet++/Locate3D, captured with an iPhone LiDAR and a clear domain shift from the training data, Qwen-3D-3B reaches 55.7 [email protected] against 32.3 for UniVLG and 33.2 for Video-3D-LLM, a gap above 22 points.
Instance segmentation (Table 3, ScanNet200): Qwen-3D-3B scores 25.3 mAP, more than double the 12.1 of Grounded-3D-LLM (the only prior generalist that attempted full-scene detection), approaching the language-prompted specialist UniVLG at 27.9. The best closed-vocabulary expert, ODIN, still leads at 31.5.
2D capability barely moves. On RefCOCO, Qwen-3D-3B scores 88.1 versus 89.1 for the Qwen2.5-VL-3B base, a 1-point drop.
Specialist 3D models have long beaten generalist ones, usually blamed on scarce data or weak backbones. This paper's answer is that the bottleneck is the language-translation step in decoding. Switch to a query-based mask decoder and move attention into 3D, and the generalist matches specialists on grounding, beats them on QA, and doubles as a full-scene segmenter that most box-only grounding models cannot be.
It is also practical. It compresses up to 400 multi-view frames into a set of 3D tokens, and cutting the input from 400 to 15 frames leaves grounding at 56 percent with no cliff. A forward pass takes about 2 seconds (20 ms to unproject, 1 ms to voxelize), plus a one-time SLAM reconstruction of 3 to 7 seconds; the reconstruction is amortized per scene, so later queries cost only the 2 seconds. Code and weights are released at qwen-3d.github.io.
The fit is scene-scale robotics, embodied AI, and AR/VR workloads that need referring, segmentation, and QA over real spaces. Anything beyond static scenes is out of scope.
The authors name three: the model handles only static scenes and would need 4D representations for dynamics; it performs a single grounding step, which limits compositional reasoning over multiple entities; and it relies on externally estimated depth and pose, so reconstruction errors cap performance.
Two more caveats stand out. First, the depth and pose come from external models (MoGE and the like). The pose and depth noise experiments on ScanRefer are encouraging, with the model holding up under heavy synthetic noise and even beating UniVLG under extreme pose misalignment. But synthetic Gaussian noise is not the same as real SLAM drift and dynamic-object artifacts, which go untested. Second, on instance segmentation the model still trails the closed-vocabulary specialist ODIN by about 6 mAP; "approaching specialists" holds for grounding and QA, less so for segmentation.