SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?
Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Wu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, Wentao Zhang, Weikang Li
cs.CV, cs.CL
2026-10-20
VLM agents explore 80 indoor scenes to place 3,140 object centers. Claude Opus 5.5 reaches 41.71% mean IoU, yet only 3.25% of targets exceed IoU 0.9.
Spatial question answering scores whether a choice is correct. Left or right, near or far, and counting all have small answer sets, so guessing, elimination, and language priors can land on the right option. A correct option is not a metric map you could walk or grasp with. Doing either requires choosing the next view, tying the same object across images, placing it in one coordinate frame, and letting that estimate decide what to look at next.
ESI-Bench and S3-Bench add active looking to question answering. Theory of Space scores discrete cognitive maps. A right answer can still sit on a wrong coordinate. SpatialScout scores the object-center map an agent builds by exploring.
The catalog is 80 indoor scenes and 3,140 targets, rendered in SAPIEN through ManiSkill3, drawn from iTHOR, RoboTHOR, ArchitecTHOR, ReplicaCAD, and ProcTHOR. Twelve scenes have 35 targets and 68 have 40. Of the targets, 1,489 sit in equivalence groups with more than one instance in the same scene, so repeated categories are easy to cross-wire. The search radius runs from 1.95 m to 9.23 m. Objects left off the inventory are removed from the rendered images.
The agent receives the inventory, the first RGB frame, intrinsics, pose, and the radius. It does not receive ground-truth centers. The frame is fixed to the initial camera: origin at the camera, +x along the initial viewing direction, +y to the left, +z up. Each submission is a center in meters and must lie within r + 0.5 m. The center is the midpoint of the tight axis-aligned box, so the target does not depend on an asset origin that often sits off the visible geometry. Articulated objects union their link geometry first. Pixels land on surfaces, while the box midpoint is usually inside the object, so a triangulated point still has to be pushed to a center using the visible extent.
All models share ten tools, and none of them returns ground truth or a score. The agent moves the camera, captures images, triangulates marked pixels from at least two frames by least squares against the viewing rays, and may submit or overwrite centers until it stops. One model call can run several tools in sequence. A low residual only means the rays meet. It does not mean the pixels are the same part, or that the point is the box center.
Each episode allows 100 model calls and 100 images, including the first frame. Frames are rendered at 512×512 and delivered with the long side at most 384 pixels. Missed targets count as zero in the all-inventory mean. 3D IoU translates the reference box onto the predicted center and then computes overlap, so the score tracks center error relative to object size. The same absolute miss is less visible on a large object than on a small one. Hit rates use a separate within-group matching at strict thresholds of 0, 0.5, and 0.9.
Seven frontier VLMs each run one episode on the same 80 scenes. Reasoning effort is xhigh, except an extra GPT-6 Astra run at medium.
| Model | All-target mean IoU | IoU>0.5 | IoU>0.9 | Submitted mean IoU |
| Claude Opus 5.5 | 41.71% | 47.23% | 3.25% | 46.69% |
| GPT-6 Astra xhigh | 33.16% | 35.15% | 1.62% | 41.16% |
| Claude Sonnet 5.5 | 32.26% | 33.60% | 1.40% | 38.99% |
| GPT-6 Astra medium | 31.18% | 31.55% | 1.27% | 37.16% |
| GPT-6 Sol | 20.83% | 18.95% | 0.48% | 28.65% |
| Qwen3.8-Max | 6.80% | 4.52% | 0 | 12.14% |
| Kimi-K3 | 4.22% | 2.77% | 0 | 6.55% |
| DeepSeek-V4.1-Flash | 3.48% | 0.92% | 0 | 5.80% |
Overlap is common and tight alignment is not. Claude Opus 5.5 has positive overlap on 77.36% of all targets, against 67.79% for GPT-6 Astra at xhigh. The share above IoU 0.9 is 3.25% for Opus and 1.62% for Astra. Among submitted targets only, those rates are 3.64% and 2.09%. Coarse layout and a fitted center are different outcomes.
Single scenes show the split. Opus submits all 35 targets in one scene, with 0.073 m center error and mean IoU 0.530. Astra, across two scenes, submits 65 of 80 targets, with 0.141 m error on the ones it submits. DeepSeek-V4.1-Flash submits all 40 targets in its example, with 1.000 m error and mean IoU 0.034. A complete list can still be a scattered map.
How the tools are used separates the models more than how many photos they take. On the same 80 xhigh episodes, Astra captures 32.41 images on average, supports 99.33% of submitted objects with a successful triangulation, draws geometric warnings on 2.69% of those calls, and has a median center error of 7.08 cm. DeepSeek captures 39.63 images, covers 63.48% of submissions with triangulation, warns on 38.65%, and has a median error of 68.17 cm. More views with bad correspondences move the center from centimeters to decimeters. Opus makes the most tool calls, costs 28.85 USD per scene, and posts the highest mean IoU. Among the open models the paper compares, only DeepSeek is cheaper than GPT-6 Sol. Qwen3.8-Max and Kimi-K3 cost more and score lower.
A separate read of 40 trajectories, ten each from Astra, Opus, DeepSeek, and Qwen3.8-Max, all at xhigh, finds six that both cover at least 90% of targets and keep matched center error at or below 0.2 m. All six ground geometry in multiple views, add missing objects as they explore, and revise centers afterward. Of the 34 that miss the bar, 26 have incomplete coverage, 21 keep a clearly wrong position even when the triangulation residual is low, and 13 stop with at least ten model calls still unused.
Question-answer accuracy does not say whether the coordinates are usable. Opus, the strongest model here, clears IoU 0.5 on 47.23% of targets, so a coarse layout is within reach. Only 3.25% clear IoU 0.9, so precise centers stay rare. Under the same budget of 100 calls and 100 images, binding submissions to triangulation and then revising them is what brings median error to 7.08 cm. Spending later calls on spinning the camera, and looking less often, does not fill the map in. The inventory, poses, and intrinsics are given. Discovering unknown objects is outside the task.
The paper has no limitations section. There is no human correspondence baseline and no score for geometric models such as DUSt3R or VGGT, so 41.71% mean IoU has no measured ceiling. The box midpoint and the visible surface are different points by construction. IoU only translates the reference box, so errors in orientation and size never show up on their own. The metric is sensitive to where the center sits relative to object scale, and silent on whether the rest of the geometry was recovered. No sensitivity check on that definition is reported.
The simulator hands over pose and intrinsics, deletes every object that is not on the inventory, and shrinks inputs to a 384-pixel long side. Each model runs once per scene, with no variance in the table. Runtime failures are dropped from the aggregate, and failure rates are not reported. The only per-scene price stated in text is Opus at 28.85 USD. Part of the trajectory write-up uses Codex driven by GPT-6 Astra to read Astra's own logs. The geometric scores are computed offline. The written failure categories are not independent of a model being graded.
Three counts in the paper do not agree with each other. The listed source pools are 32, 12, 10, 6, and 30 scenes, which add to 90, while the benchmark is described as 80 scenes. Twelve scenes of 35 targets plus 68 of 40 add to 3,140 targets, but one sentence says 3,410. Astra's submitted mean IoU is 41.16% in the table and 41.76% in the prose.