Pittsburgh researchers find 3D spatial maps in VLM middle layers, boosting spatial reasoning 12.1 points

机器之心 · wechat · 2026-10-20

Researchers at the University of Pittsburgh, Haoming Wang and Wei Gao, look inside VLM latent spaces to ask whether models form 3D spatial structure with adjacency relations, not just answer questions correctly. The paper is a NeurIPS 2026 spotlight.

Method:

Object masks from synthetic scenes aggregate visual tokens by coverage and average across time to get object-level representations.
Averaging the same object identity across scenes cancels positional variation, estimating stable identity attributes like color and shape; SVD finds those directions and they are projected out, leaving position-related variation.
PCA projects the separated representations to 3D for side-by-side comparison with true layouts.

Findings: Middle layers retain scene structure more clearly, with object arrangement and adjacency better matching reality; earlier layers lean visual, later layers serve language output. A Dirichlet energy test shows lower energy ratios in middle layers, more so after removing appearance. The team proves that minimizing Dirichlet energy yields Laplacian eigenmaps after mean-centering and PCA, grounding the visualization. Perturbing along space-related directions shifts coordinate probe readings directionally, while control directions do not.

Application: Adding Dirichlet energy to the fine-tuning objective with 500 LoRA steps at selected middle layers raises Qwen2.5-VL-7B accuracy on VSI-Bench simple relative direction from 47.9% with plain LoRA to 60.0%, a 12.1-point gain.

Original post →

Pittsburgh researchers find 3D spatial maps in VLM middle layers, boosting spatial reasoning 12.1 points

More from Research