Imagine3D-LLM: a 7B model that imagines 3D scenes before answering beats 72B rivals by 29 pts

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han, Mungyeom Kim, Minkyeong Jeon, Heeseong Shin, Wonjun Moon, Federico Tombari, Daniel Barath, Marc Pollefeys, Seungryong Kim, Sunghwan Hong

NeurIPS 2026

cs.CV, cs.CL

2026-09-30

Learnable Gaussian summary tokens let a 7B MLLM reconstruct multi-view scenes as compact 3D Gaussians before answering, hitting 68.5 on SPAR-Bench, 29 points above a 72B model.

What problem this solves

Multimodal LLMs (MLLMs) are strong on a single image and weak on a pile of views of the same scene. Ask a question that requires knowing where the camera sits relative to the furniture, and even frontier models fall well short of human performance. Two fixes dominate the literature: stitching 3D point coordinates or bird's-eye-view markers into patch features, or fusing in features from reconstruction foundation models like VGGT and CUT3R. Both feed the model pixel-level geometry, and both have been delivering shrinking returns.

The KAIST AI and ETH Zürich team starts from a different observation. Cognitive science says humans do not keep pixel-accurate correspondences in their heads. Given several views, people recognize the same objects across them, infer rough relative camera geometry, and assemble a coarse 3D layout. The representation behind human spatial reasoning is object-level and approximate. So the question becomes: instead of telling the model the geometry, can it be made to imagine the scene?

Method

Imagine3D-LLM builds on LLaVA-Video-7B. The change sits in the input sequence: 2,592 learnable Gaussian summary tokens inserted between the image tokens and the text tokens. With 32 input images producing 6,720 image tokens, that is roughly 81 summary tokens per image. Training combines three objectives:

• From layer 14 of the 28-layer LLM, the hidden states at the summary token positions pass through a three-layer MLP and decode into 3D Gaussian Splatting primitives, 32 Gaussians per token. These are rendered back at the input viewpoints and compared against the real frames with a photometric loss (MSE plus LPIPS).
• Text tokens get standard next-token prediction, conditioned on both image and summary tokens.
• Pure photometric supervision converges too slowly to be practical inside MLLM finetuning (feed-forward 3DGS estimators typically train for hundreds of thousands of iterations), so a frozen compact Gaussian teacher, ZipSplat, distills into the summary tokens: cosine alignment at the token level plus an L2 loss on Gaussian parameters. The teacher is dropped at inference.

Two design choices carry the paper. First, Gaussians are predicted from dedicated summary tokens rather than directly from image features, which blocks the shortcut of copy-pasting 2D content into image-aligned Gaussians. Second, the summary tokens are far fewer than the image tokens (2,592 versus 6,720), an information bottleneck: an object appearing in many views has to collapse into a shared token, so the model must work out which objects recur and how they fit together in 3D. K-means over the trained summary tokens clusters them by object, with no clustering supervision anywhere.

The ablation nails down the causality: feeding the teacher's query tokens straight into the LLM scores 56.4 on SQA3D EM, distillation without reconstruction 56.6, both sitting at the baseline's 56.5. Only the reconstruction loss moves the number, to 63.8.

Results

SPAR-Bench splits into low, medium, and high cognitive levels, and the gains concentrate in medium (view-change inference) and high (spatial imagination). A controlled baseline sharing the same backbone, data, and schedule goes from 51.5 to 67.0 on the medium split (+15.5), so the gain traces to the objective rather than the data. On average, the 7B model beats Qwen2.5-VL-72B by 29.1 points.

ScanQA EM is one metric it does not win (29.9 against Ross3D's 30.8). Dropping distillation and training reconstruction-only for 4 epochs reaches 63.7/67.9 versus the full model's 63.8/68.5 at 1 epoch, so distillation mostly buys training time. The token count has a sweet spot: 1,296 is too few, 5,184 loosens the bottleneck, and both underperform 2,592.

Why it matters

Table 7 is the actionable finding for anyone building 3D MLLMs: handing the LLM features from a 3D foundation model does essentially nothing, while making the LLM compute the 3D structure itself does. That is a direct counterpoint to the passive-geometry line of positional embeddings, visual markers, and feature fusion. And although reconstruction supervision touches only the summary tokens, the LLM's own image features come out more view-consistent, with sharper cross-view attention and object-consistent PCA structure. A structured objective reshapes representations across the stack. Inference cost is modest: 2,592 extra tokens, no teacher, no point cloud input. For general MLLM teams this is a portable training objective, not a bespoke architecture.

Limitations

• No dedicated limitations section. The authors' own admission: photometric-only supervision is too slow to converge inside MLLM finetuning and needs the ZipSplat teacher, which ties the 3D prior to one specific model.
• Six of the seven benchmarks come from ScanNet, all indoor. Generalization to outdoor or in-the-wild multi-view inputs is untested. The authors cite the known concern about ScanNet overfitting and add a 100K subset of SPAR-7M to counter it.
• Margins over Ross3D are thin on some benchmarks (SQA3D +0.8), and ScanQA EM is lost outright; the large gaps are on SPAR-Bench and Scan2Cap.
• The 'imagined' artifact is a set of Gaussians whose rendering quality is never quantified, so how accurate the mental reconstruction is stays open; the analysis stops at attention maps and PCA visualizations.
• Training used 16 GH200 GPUs, a nontrivial reproduction bar.

Terms

• MLLM: a multimodal LLM that takes images and text together.
• 3D Gaussian Splatting (3DGS): a scene representation as colored, semi-transparent 3D Gaussians that can be differentiably rendered from any viewpoint.
• Gaussian summary token: a learnable placeholder token inserted into the input sequence that learns to pool multi-view information.
• SPAR-Bench: a benchmark scoring 3D perception and reasoning at low, medium, and high cognitive levels, built to punish language shortcuts.
• Pretext task: an auxiliary training task that does not match the downstream task but forces useful representations.
• LPIPS: a perceptual image-similarity metric commonly used inside reconstruction losses.

Source

• arXiv 2609.38177v1
• PDF
• Hugging Face
• GitHub · https://github.com/cvlab-kaist/Imagine3D-LLM

Related papers

• DC-SAE: a dual-stream semantic-plus-pixel autoencoder cuts gFID 55% vs DC-AE at 32x compression
• Looped Diffusion Transformer: a 260M text-to-image model beats a 6.5x larger one at 4.9x less compute
• ThinkV2V makes the MLLM think before editing: a 5B DiT beats 14B baselines on implicit instructions
• VGGT geometry routed into a video diffusion prior wins sparse-view NVS with just 1K scenes
• PreviewDiff: a multimodal critic critiques mid-denoise previews and branches the search, beating budget-matched Best-of-N
• LeRF: VLMs learn to draw reference coordinate frames on images for +11 pp perspective-taking gains

All paper explainers