Zhejiang team's Spatial-Interactor teaches VLMs spatial reasoning from interaction traces
青稞AI · wechat · 2026-10-17
Zhejiang University's OmniAI lab and SAP Singapore propose Spatial-Interactor, teaching vision-language models spatial reasoning from interaction trajectories — addressing that recognizing each frame doesn't mean understanding a continuously changing world.
The method uses a three-tier curriculum: L1 external object changes, L2 egocentric viewpoint changes, L3 integrating full camera trajectories. An automated pipeline turns simulation and real interaction records into LSI-108K, 108K verifiable spatial QA pairs. Training combines SFT with On-Policy Distillation, where a teacher guides the student along its own generated reasoning using segmented motion descriptions.
Evaluated on Qwen2.5-VL and Qwen3-VL bases across 8 spatial benchmarks: the 7B model improves VSI-Bench 37.9→49.0, multi-view SPBench-MV 34.0→67.6, and SAT-Real 54.7→74.0, with gains transferring to held-out benchmarks. On embodied tasks, WalkerBench success doubles (7.0%→14.0%). Paper, code, and data are open-sourced.
More from Research
- Laude Institute launches FrontierCS 2: agents must beat deleted paper contributions blind — lateinteraction · 2026-10-23
- RFD4-Proteina's 'underrated' docs detail partial diffusion for protein redesign — AllThingsApx · 2026-10-23
- MPI-Sec PhD Thesis on Vulnerability Detection Benchmarks Includes Pre-LLM Classic "Top Score on the Wrong Exam" — mboehme_ · 2026-10-23
- LiveNerf's 29-day data finds no nerfing of Claude Opus 5.5 — TheOnlyVibemaster · 2026-10-23
- New paper makes 9 falsifiable predictions toward a major AI-enabled terror attack by 2027-2028 — StephenLCasper · 2026-10-23
- HumanoidPose recovers humanoid robots' full-body joint states from RGB video alone — cholz · 2026-10-23