VLMs hide spatial structure in their representations, and training can preserve it
jiqizhixin · x · 2026-10-30
Researchers Haoming Wang and Wei Gao at the University of Pittsburgh (NeurIPS 2026 Spotlight) show that spatial structure persists inside vision-language model representations even when object appearance dominates the signal — and that training can preserve it.
- Separate appearance from layout: pooling object features across frames and varied arrangements, then projecting out dominant appearance directions, reveals object relationships hidden in the original features.
- Keep neighbors close: during LoRA training on synthetic 3D coordinate tasks, a Dirichlet-energy regularizer shapes an intermediate layer to reward preserving spatial neighborhoods rather than just fitting final answers.
- Measured downstream gains: the approach improves performance on spatial benchmarks (VSI-Bench), recovering scene topology from egocentric visual input via latent-representation probing.
More from Models
- DeepSeek V4.1 Flash tops open-model usage among research agents, index data shows — AkariAsai · 2026-10-30
- Dev claims Gemini Argon makes 'researcher-level jumps' other models miss as 'carbon' looms — vitaliychiley · 2026-10-30
- Meta's Alexandr Wang touts Muse: 6.7x more concise than ChatGPT at answering the same question — alexandr_wang · 2026-10-30
- OpenAI's subsidy shift rewrites the score-vs-cost Pareto frontier, analysis finds — RexDouglass · 2026-10-30
- Phonic tops VAmoS benchmark for agentic speech-to-speech models — graceisford · 2026-10-30
- DeepSeek Frontend Leaks cyber-v3/v4 Keys; Still Hiring Attackers as V4 Excels at Security — teortaxesTex · 2026-10-30