SpatialSpeak: QA-Native Reconstruction with Local and Global Context for Spatial Chain-of-Thought Reasoning
Yang Cao, Jiaxin Zhang, Dave Zhenyu Chen, Yingji Zhong, Ruiyuan Gao, Lanqing Hong, Dan Xu
cs.CV, cs.AI
2026-09-27
A 4B VLM pretrained on QA-form reconstruction then spatial CoT scores 62.8 on ReVSI, +8.7 vs SpatialStack-4B; CoT-VC gain rises from 2.6 to 6.9 with QA-RP.
Multi-view VLMs already ingest geometry from frozen reconstructors such as VGGT, then train on spatial QA with answer-only labels. That puts 3D structure into hidden features. It never asks the model to write down a bed's center and size, then compute the longest side from those numbers. Quantitative questions need both a decent geometric estimate and a correct use of that estimate. Answer-only SFT leaves both steps implicit.
SpatialSpeak, from HKUST, Huawei Noah's Ark Lab, and Harbin Institute of Technology, tests a simple bet: teach the VLM to speak local point geometry and global object layout through the same autoregressive interface used later for reasoning. The baseline is already VG-LLM-style fusion with a frozen VGGT encoder, a Transformer that turns multi-view frames into geometric features. What is missing is QA-form reconstruction pretraining and explicit geometric CoT, meaning chain-of-thought that writes the numbers down.
Both stages train only the LLM. The vision encoder, VGGT geometry encoder, and MLP projector stay frozen. The backbone is Qwen3-VL-4B with 32 frames per clip. Every 3D coordinate is expressed in the first-frame camera frame, so points and objects share one origin.
Stage I is QA-RP. Two text QA tasks share a next-token loss:
There is no extra 3D head and no geometric regression loss. Reconstruction is forced through the same text channel the CoT will use.
Stage II is CoT-VC. Deterministic generators on ScanNet boxes write traces for four question types: distance, size, count, and closest object. Distance is center-to-center minus two approximate radii from axis-aligned boxes; size is the longest side; count is the instance tally; closest object scores each multiple-choice option. A relative-error threshold of 0.3 labels the geometric answer HIGH or LOW. HIGH keeps the derived answer. LOW keeps the 3D chain, inserts a visual-check note, and supervises the ground-truth answer. That fallback is Visual Compensation. Tasks without a template, such as relative direction, stay on direct-answer labels.
Stage I runs 1 epoch on 8 H800 GPUs, global batch 32, learning rate 5e-6. Stage II uses batch 64 and 1e-5, on the LLaVA-Hound split of LLaVA-Video-178K plus SPAR-7M, mixed with the constructed CoT data.
ReVSI, a spatial benchmark that fixes VSI-Bench labels and answer-distribution bias, isolates the two stages. Neither component: 52.4. QA-RP only: 55.9. CoT-VC only: 55.0. Both: 62.8. CoT-VC is worth 2.6 points without QA-RP and 6.9 with it. Dropping both local and global reconstruction queries returns 55.0; dropping only global queries yields 59.9, only local 59.3. Visual Compensation itself is 4.3 points (58.5 without it, 62.8 with it). Setting the reliability threshold to 0.1 or 0.5 drops the score to 59.6 and 59.5, both below 62.8 at 0.3.
Main comparisons, ReVSI under a shared 32-frame protocol:
| Method | ReVSI | VSI-Bench (normal) | VSI-Bench (scaled) | SPAR-Bench |
| SpatialSpeak-4B | 62.8 | 63.3 | 73.0 | 76.0 |
| SpatialStack-4B | 54.1 | n/a | 67.5 | 72.0 |
| GeoThinker-8B | 54.0 | n/a | 72.6 | 68.2 |
| Omni-View-7B | 48.4 | 55.4 | n/a | n/a |
Against SpatialStack-4B on ReVSI, object counting is 64.9 vs 36.3, room size 66.0 vs 54.4, relative distance 71.9 vs 62.9. Five of seven categories are best in the table. With scaled data (SPAR, VLM-3R, VSI-590K plus CoT), the 4B model reaches 73.0 on VSI-Bench, 0.4 above GeoThinker-8B at 72.6.
In qualitative examples, a trash-bin-to-toilet closest-point estimate is 3.6 m against a 3.5 m ground truth; the variant with neither stage answers 2.7. Chair count is 4 vs 6.
On 200 held-out ScanNet scenes, unaligned point-map Acc/Comp is 8.9/9.3 cm vs CUT3R 13.7/12.7 and MapAnything 36.3/28.4. After ground-truth Sim(3) alignment, which allows scale, rotation, and translation, the scores become 5.0/5.0, a bit worse than CUT3R at 4.7/4.6. Reconstruction here is a substrate for reasoning, not a new specialist reconstructor.
Fused geometric tokens are not enough if the model never has to utter the numbers it reasons with. Casting reconstruction as QA keeps the implementation cheap: no new 3D head. A 4B LLM-only finetune with a public project page is easy to try on indoor multi-view spatial QA.
This is still an incremental move on the VG-LLM line. What it adds is two stages of text supervision. The jump in counting matches the global object-center task, which trains the model to list chairs.
There is no Limitations section in the paper. Several gaps are easy to see.
CoT templates cover only distance, size, count, and closest object. Relative direction stays answer-only. LOW targets copy the ground-truth answer, so the model may learn to switch answers when unsure, not to re-inspect the frames. Reliability labels are computed from ground truth at train time; at test time the model must self-tag HIGH or LOW, and that calibration is not reported.
Labels depend on ScanNet depth and extrinsics: indoor, metric, cameras known. The vision tower and VGGT stay frozen, so geometric skill is still capped by that prior. After alignment, reconstruction trails CUT3R, which fits a method that writes usable metric numbers into the LLM rather than beating specialist reconstructors. Some ReVSI baselines are copied from the original paper, others re-run at 32 frames, so the comparison is not fully matched.