Show-Harness: Just a VLM Agent Can Play Robots
Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, Mike Zheng Shou
cs.RO, cs.AI, cs.CV, cs.MM
2026-09-10
Show-Harness lets VLMs drive robots with discrete semantic actions. Gemini-3.1 Pro hits 89% zero-shot on 10 real tasks; a 2B VLM reaches 86% after LoRA, versus 39% for π0.5.
Foundation VLMs already parse objects, spatial relations, and long-horizon goals, yet that knowledge does not drive a robot arm. VLAs fine-tune the backbone to regress continuous actions, collapsing semantics into an opaque pixel-to-motor map that usually needs new data per task, scene, or body. Hierarchical agents take the other exit: the model emits subtasks or API programs, and a downstream controller owns physics, so the VLM never sees how intent is realized.
Show Lab argues the missing piece is the interface. Show-Harness compresses control into discrete semantic units: forward/back/left/right/up/down relative to a reference view, axis-wise rotation, grasp, release, done. Each step is small and observable, so the VLM stays responsible for fine physical choices. Impedance control, inverse kinematics, and workspace limits stay inside an embodiment-specific interpreter.
At each step the harness packs multi-view images and proprioception into context. Optional plugins handle subtask planning, failure recovery, action chunking, and adaptive step size. The VLM emits one semantic unit. The interpreter turns it into a bounded Cartesian increment. Franka tracks with impedance control, AgileX streams inverse-kinematics joint targets, and the simulator runs operational-space commands. A new body needs a new interpreter, not a new action vocabulary.
The same interface supports two modes. Zero-shot: a closed-source frontier VLM drives the robot with no fine-tuning. Default is Gemini-3.1 Pro at medium thinking effort, 2 cm near the target and 4 cm farther out. Fine-tune: Qwen3.5-2B gets rank-64 LoRA on language-model linear layers, about 3% of parameters, and predicts units in its native vocabulary with no action head. Demonstrations come from GUMI, a GUI where humans and agents press the same semantic keys, without teleoperation hardware. Real-robot data is 164 episodes and 7.8K decision steps, mixed across Franka and AgileX. Sim-to-real uses 230 simulated episodes collected the same way.
Ten object–receptacle tasks, 10 trials each, randomized placements, 50-step timeout as failure. Block, banana, and tennis ball appear in demos; teddy and chess piece are held out for the fine-tuned model.
| Setting | π0.5 | G-VLA | Show-Harness ZS | Show-Harness FT |
| Cross-task | 39% | 57% | 89% | 86% |
| Cross-environment | 40% | 65% | 100% | 88% |
| Cross-embodiment | 41% | 52% | 93% | 87% |
| Sim-to-real | 0/20 | / | / | 13/20 |
Zero-shot embodiment transfer is interpreter-only: 48/50 on Franka, 45/50 on AgileX. The code-as-policy baseline CaP-X scores 13% cross-task. Shrinking the interpreter step from 2 cm to 1 cm, with no retraining, lifts fine-grained tasks from 60% to 82% (ZS) and 40% to 65% (FT); π0.5 scores 18% on the same demos and reaches 62% only after extra fine-grained training. Dropping subtask planning cuts success to 60%, often dragging objects without lifting. Multi-view guidance moves success from about 58% to 96% versus a global view alone. On reasoning-heavy tasks, ZS with situated planning hits 85%; FT and π0.5 alone score 10% and 0%.
For embodied work the implication is concrete: do not rush to turn a VLM into a motor regressor. A vocabulary that models can reason over, with deterministic grounding, lets closed-source upgrades become robot upgrades, and lets a 2B model share the same control after a few GPU-hours. GUMI also drops demonstration collection from a teleop desk to a keyboard and a browser, with trajectories reusable across interpreters.
That does not make VLAs obsolete. Continuous actions may still win on contact-rich, high-speed trajectories. What Show-Harness shows is that on today's pick-and-place, cross-scene, and cross-arm evaluations, the interface can move success more than model scale.
Evaluation is almost entirely parallel-jaw single- and dual-arm work. The paper says so: humanoids and dexterous hands are future work, and there is no tactile or force sensing. The ten main tasks are one family of pick-and-place, with object–receptacle pairs carrying the diversity. Zero-shot defaults to Gemini-3.1 Pro, so API drift becomes robot drift. The fine-tuned policy uses a stripped context that drops many plugins, so 89% versus 86% is not a clean head-to-head. Sim-to-real at 13/20 transfers, but is not deployment-ready. Ten trials per task is a small sample. Interpreter step size and safety bounds still need human calibration; a new table height or gripper is not free.