VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control
Zhongbo Zhang, Jiayi Jin, Yifan Wang, Zaibin Zhang, Haiwen Diao, Lijun Wang, Huchuan Lu
cs.RO, cs.CV
2026-09-17
VA-Bench has MLLMs learn RGB demos, pick views, and issue metric Cartesian commands. Best success is 53.93±3.17%; active cameras beat passive five-view input by 29.64 points.
Most spatial benchmarks stop at describing a picture. A model that can name left-right relations on fixed views still fails when a grasp face is occluded and a pixel shift does not equal millimeters in world coordinates.
Robot suites often supply object poses, skill primitives, or a learned action head. They test policy execution, not whether a general-purpose MLLM can turn a spatial judgment into a number the arm can run. VA-Bench welds those pieces into one loop: choose a view, estimate geometry, issue a Cartesian command, then revise from execution feedback.
The environment is RoboTwin physics simulation. Fourteen task families, 11 single-arm and 3 dual-arm, cover grasping, placement, tool contact, and bimanual coordination, with 20 physically verified seeds each. Seven held-out variants change geometry or layout. A separate track asks for five-object composition.
Models see RGB, filtered proprioception, and a textual procedure they wrote from an RGB-only success video. The video carries no trajectories, poses, or contact labels. Robot actions are parameterized: 1–100 mm world-axis translations, 1–90 degree gripper-local rotations, and gripper aperture. Camera actions pick gripper-centered semantic viewpoints plus bounded local tweaks, and they share the episode budget with the arm. A fixed, model-agnostic inverse-kinematics layer executes only the targets the model names. There is no learned action head.
Primary score is the physics checker's terminal success, averaged over three independent runs. Nine trajectory diagnostics, majority-voted by three annotators, locate failures in perception, planning, or recovery.
Qwen3.8-max leads at 53.93±3.17% macro-average over 14 tasks. Opus-5 sits at 52.86±2.79% and GPT-5.6-sol at 51.55±4.31%; the three-run ranges overlap, so the ranking among them is not reliable. Qwen reaches 65.61% on single-arm tasks; the best dual-arm mean is GPT-5.6-sol at 22.22%. Below the top three the floor drops: GPT-5.6-terra is 23.10%, everyone else is at or below 15%.
Diagnostics are harsher. In the annotated run the strongest models hit 100.0% target localization and at least 99.6% manipulation semantics, but only 78.9% spatial relations. They find the object and still mis-estimate robot-object geometry. Online correction is 46.7% for Qwen and 32.4% for Opus-5. Annotators agreed unanimously on 95.7% of applicable labels (Fleiss κ 0.942).
Active cameras lift Qwen from 27.86% with a passive five-view bundle to 57.50%. Placement collapses without view control: placecubeinbowl falls from 85% to 5%. Geometric transfer costs GPT-5.6-sol 32.14 points from an 80.00% matched base; Qwen drops 10 points to 67.86%. No model finishes a strict five-object episode. The best partial score is 61/100 placements.
| Setting | Success | Contrast |
| Qwen3.8-max, active camera | 57.50% | same model, passive five-view 27.86% |
| Qwen3.8-max, three-run macro | 53.93±3.17% | localization diagnostic 100.0% |
| GPT-5.6-sol, held-out geometry | 47.86% | matched base 80.00% |
Demonstration summaries do not transfer by author strength. Qwen's own summary costs the other four models 31.25 points on average, 51.67 for GPT-5.6-sol. The Opus-5 summary helps weaker readers by 7.50 points on average.
For people building spatial or embodied MLLMs, pointing at the right object is not the same as putting a cube in a bowl. Choosing where to look beats dumping five views at once: the missing skill is task-conditioned evidence, not coverage. This is a benchmark, not a new controller. Local 27B/31B models still near-zero on placement and dual-arm work even with a strong summary.
Simulation, a virtual camera, and a frozen executor are not a real robot. The paper leaves the active-versus-passive mechanism open: view relevance, cross-view fusion, or both. Dual-arm and long-horizon completion barely move. The top three models are statistically tied. Summary transfer uses three tasks and one run. Small-model wins concentrate on grasping, so the interface remains hard for weaker backbones.