SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation
Jinsheng Quan, Jianhua Li, Siyi Xie, Xuanke Shi, Kewang Deng, Zukai Chen, Feifei Shao, Lei Yang, Quan Wang, Yawei Luo
cs.CV, cs.AI
2026-08-14
Zhejiang University and SenseTime cast 3D reconstruction, dense correspondence, and spatial reasoning as instruction-conditioned generation in one Bagel-7B native multimodal model: first-place averages on all four spatial-reasoning benchmarks, 66.60 vs Spatial-MLLM's 33.74 on SPAR.
Recovering geometric structure, establishing cross-frame correspondences, and understanding spatial relations are three complementary views of one physical scene, yet they live in three separate systems: dedicated reconstruction networks, separate optical-flow stacks, and VLMs with bolted-on geometric modules. Representations do not transfer. Prior unification attempts either added task-specific regression heads or external geometry predictors, never exploiting a native multimodal generative model.
SPARGen's core move translates heterogeneous spatial outputs into the two native output modes of a multimodal generative model, built on Bagel (7B, MoT backbone with understanding and generation experts sharing joint attention):
The instruction is the unified interface: the same model switches from reconstruction to flow to QA by changing one sentence, with no task-specific heads or external modules. Training spans spatial QA (MindCube, OmniSpatial, OST-Bench, SPAR-7M), visual geometry (fifteen sources including ScanNet++, Hypersim, CO3D), and optical flow (FlyingThings3D, Spring), fine-tuned 100K steps on 64 H100s with the VAE frozen.
| Benchmark | SPARGen-7B | Strongest comparison |
| SPAR average | 66.60 | Spatial-MLLM-7B 33.74; Qwen2.5-VL-72B 36.84 |
| MindCube average | 76.04 | Spatial-MLLM-7B 66.19 |
| OST average | 79.16 | Spatial-MLLM-7B 74.80 |
| KITTI flow EPE (zero-shot) | 4.09 | RAFT 5.03; FlowFormer 4.10 |
| Sintel depth AbsRel | 0.235 | VGGT 0.265 |
| CO3D camera rotation RRA@30 | 96.84 | VGGT 98.79 |
First place in 13 of 15 non-proprietary categories. The geometry side narrows but does not sweep: VGGT still leads camera pose and 7Scenes reconstruction accuracy. For context, GPT-4o averages 41.52 on SPAR and Claude Sonnet 4.6 44.19, both below this 7B model.
Ablations confirm the three supervisions feed each other: removing geometry supervision drops both flow and reasoning; removing flow supervision raises 7Scenes errors and drops SPAR from 66.60 to 65.92; removing reasoning supervision slightly hurts reconstruction while slightly helping flow, indicating a mild capacity competition between autoregressive token generation and dense-field prediction.
Spatial intelligence is the foundation of embodied AI and robotics, and it is exactly where current VLMs are weakest. SPARGen demonstrates that the two native pathways of a multimodal generative model (autoregressive sequences and rectified-flow image generation) can carry heterogeneous spatial tasks without external geometry modules, and that multi-task supervision feeds back: geometry helps reasoning, correspondence helps reconstruction. The practical value is one model replacing three: a single checkpoint for depth, flow, and spatial QA, cutting deployment and iteration cost. There is a genuine incremental component, as geometry metrics do not uniformly beat specialized models, but "unified without falling behind" had not been achieved before.
The authors state one: frozen-VAE spatial compression is a bottleneck for geometric edges and high-precision physical quantities. Depth passes through an 8x-downsampled latent before reconstruction, so sub-pixel edges are bound to blur.
A close read adds more. The 10^-3 quantization resolution caps camera-pose precision at the millimeter / milliradian level, which may fall short for high-precision reconstruction. The "13 of 15 first places" compares against a paper-selected set of non-proprietary models; GPT-4o and Claude Sonnet 4.6 appear in the table but are excluded from ranking, and neither is a spatial-task specialist. Ablations only subtract supervision types, leaving no direct test of the central claim of native generation versus simple multi-task heads, so whether the win comes from the generative pathways or the data mixture cannot be separated. And the spatial-QA benchmarks are largely LLM-synthesized or synthetic-scene based, with the gap to real physical scenes undiscussed.