Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World
Kaixiang Yao, Xu Wang, Miao Pan, Hu Xiyue, Weishi Wang, Daniel Dahlmeier, Jintao Chen, Yongliang Shen, Xuhong Zhang, Wenqi Zhang
cs.AI, cs.LG
2026-09-19
Spatial-Interactor trains VLMs on LSI-108K interaction data: SFT for local transitions, OPD for long-horizon integration. Overall rises 16.4–25.0 points; the 8B model reaches 65.9.
VLMs caption and watch video well enough; in physical space they lose state. They mix up object motion with ego-motion and scramble earlier observations on long trajectories. Most spatial training is static QA about attributes and left/right relations. It rarely supervises how a state changes after an action.
Two diagnostics size the gap. Shuffle the 32 VSTI-Bench frames and Qwen2.5-VL-3B/7B drop only 0.9 and 0.8 points, so temporal order is barely used. On local SAT-Real interaction the 7B scores 54.7; on VSTI camera displacement it falls to 8.6. GPT-5.5 falls from 88.7 to 23.9. Adding segment-level transition text lifts GPT-5.5 from 23.9 to 35.5. Local perception is not the same as integrating a trajectory.
Spatial-Interactor writes each interaction as observation, action, next observation. A three-level curriculum matches three supervision targets.
LSI-108K turns simulated rollouts (ProcTHOR, Habitat, AI2-THOR) and real trajectories (ScanNet, ARKitScenes, DROID) into about 108K verifiable QA pairs. Metadata sets the answer, templates write the question, and a frozen VLM only filters unanswerable items.
Training is two stages. SFT on 82,596 L1–L2 items plus 80K public spatial QA, visual encoder frozen. Then On-Policy Distillation on 10,712 RoomTour and 10,783 long VSTI items: 32 frames split into four segments, a frozen strong VLM writes question-agnostic transition text for the teacher only. The student samples its own chain-of-thought; the teacher scores the same prefixes with the privileged trace via KL on reasoning tokens. Answers still use verifiable rewards. The GRPO control matches everything except privileged conditioning and process distillation.
Overall rises 16.4–25.0 points across four backbones. The 8B model reaches 65.9, with MindCube moving from 29.4 to 88.4. Camera displacement for the 7B goes from 8.6 to 36.6. On held-out MMSI, ViewSpatial, SAT-Real, and SAT-Syn, gains are 4.3–10.6, mostly from SFT, with OPD adding another 1.0–1.4.
| Model | Base Overall | Spatial-Interactor | Gain |
| Qwen2.5-VL-3B | 33.9 | 55.6 | +21.7 |
| Qwen2.5-VL-7B | 35.1 | 60.1 | +25.0 |
| Qwen3-VL-4B | 47.3 | 63.7 | +16.4 |
| Qwen3-VL-8B | 48.4 | 65.9 | +17.5 |
Ablations on the 7B: public spatial SFT reaches 56.0 overall, adding L1–L2 interaction 58.4, GRPO 59.2, OPD 60.1. Versus GRPO, OPD is +5.1 on route planning and +3.1 on camera displacement. After training, shuffling 32 frames drops the 3B/7B by 6.6 and 7.0 points; the bases barely move. Closed-loop WalkerBench success goes from 7.0 to 14.0 and mean steps from 17.71 to 6.52; ESI-Bench from 32.9 to 41.2.
Spatial VLMs are short on supervision of how actions change observations and how observations update state. Interaction trajectories carry that triple for free. OPD is equally concrete: long-horizon tasks need process signal on prefixes the student actually visits, using segment text the student never sees at test time. For embodied post-training, L1 then L2 then L3 is a recipe one can retell.
The closed-loop absolute numbers stay low: WalkerBench 14.0 overall, navigation 10.0. Stronger spatial QA is not yet a reliable walker.
The privileged trace exists only in training and inherits errors from the frozen annotator. L3 is about 21K items, much smaller than L1–L2. The main tables include VSI, MindCube, and VSTI, which also contribute public samples to SFT; the held-out suite is the cleaner read. WalkerBench and ESI use subsets. The authors flag adaptive segmentation and more diverse real interaction. The visual encoder stays frozen, so geometric detail into the language tower is capped.