GST-Bench: Can VLMs Develop Global Spatial Awareness from Video?
Qifeng Zhang, Kaixiang Huang, Heng Dong, Huang Fang, Junting Chen, Junjie Zhu, Yonghang Chen, Zhiyu Zhang, Wei Li
cs.CV
2026-08-06
GST-Bench tests whether VLMs can build globally consistent spatial maps from long egocentric video; the best model scores 42.68 versus humans' 79.08, and targeted fine-tuning lifts Qwen3-VL-8B from 25.89 to 53.52.
A home robot sent to fetch an item has to build a mental map as it moves, remembering where objects sit and how they relate even when out of view. Humans do this effortlessly; current VLMs cannot. Existing spatial benchmarks mostly stare at single images or two to three views and ask whether A is left or right of B, which tests local perception. GST-Bench targets the other thing: after watching a long exploration video, can a model assemble a globally consistent layout of the whole room? ByteDance Seed built the benchmark to force exactly that.
The benchmark is organized around three questions: where am I (self localization), where is the target (object localization), and what does the scene look like (scene structure), split into 12 subtasks. Several choices are deliberately anti-shortcut. In every object-localization task the target is invisible from the query viewpoint, so the model must rely on memory of the video. The query view is sampled from a point off the video trajectory, blocking frame-matching. Scoring uses precise numbers such as distances, angles, and points on a top-down map, not coarse left/right categories. Data is generated automatically in OmniGibson with BEHAVIOR-1K across 50 indoor scenes, yielding 6,790 minutes of video and 2,762 human-verified questions.
| Model | GST-Bench avg | Note |
| Human | 79.08 | 20 q/task |
| Gemini-3-Pro | 42.68 | best zero-shot |
| Gemini-2.5-Pro | 40.95 | |
| GPT-5 | 40.85 | |
| InternVL3.5-38B | 30.71 | best open-source |
| Random guessing | 20.01 | |
| Qwen3-VL-8B (fine-tuned) | 53.52 | via GST-Train |
The gap is pervasive and large. The best model, Gemini-3-Pro, trails humans by 36.4 points; on orientation estimation the best proprietary model reaches only 21.52 versus 85.00 for humans. Open-source models hug the random line, with nine open and embodied models within three points of it. A counterintuitive result: embodied-tuned models like RoboBrain and Cosmos-Reason2 do not beat same-size general models; they inherit the same global-reasoning blind spot.
It separates "sees locally" from "integrates globally." In a control where the target is made visible, a single-image task, Gemini-3-Pro jumps from 22 to 61 on the direction task. The proprietary bottleneck is cross-frame integration, not single-image perception; open-source models fail at both. For robotics practitioners this is a direct diagnostic: most of the 36-point gap to humans comes from failing to remember and align, not from failing to see. Fine-tuning Qwen3-VL-8B on GST-Train lifts it from 25.89 to 53.52, beating every zero-shot proprietary model, but still 25 points short of humans.
All data is simulated, with a gap to real-world lighting, occlusion, and noise; top-down inputs are often unavailable at deployment. The human baseline uses only 20 questions per task. The authors concede fine-tuning narrows but does not close the gap. One underexamined point: whether embodied models score low because of training recipes that optimize local affordance over long-horizon memory, or because global god-view reasoning is simply not what they are built for. The paper only speculates.