NUS Show-Harness: A Semantic Interface Lets VLMs Control Real Robots Zero-Shot
机器之心 · wechat · 2026-09-27
NUS ShowLab's Show-Harness skips training bigger robot policies and instead inserts an embodied harness — a semantic action interface — between foundation models and robots: VLMs reason over readable units (move forward/left, rotate, grasp, release) while per-robot interpreters map them to bounded low-level commands, enabling cross-embodiment by swapping interpreters only. Decisions run in a perceive–reason–act loop rather than blind long-horizon plans.
Real-robot results: zero-shot frontier VLMs hit 89% (cross-task), 100% (cross-environment), 93% (cross-embodiment, Franka + AgileX incl. dual-arm) vs. baselines at 57–65%. A lightly fine-tuned Qwen3.5-2B (rank-64 LoRA, 3% params) transfers 230 sim demos to 13/20 real Franka tasks where trainable VLA baselines scored 0/20. An ablation shows semantic names alone keep success at 90–95%, but removing both names and physical conventions crashes it to 5% — stable symbol-to-effect grounding is the key.
The companion GUMI GUI renders the same action units for humans (keyboard teleop) and Computer-Use agents alike, while logging (observation, action) data for training. Paper, code, models, and data are open-sourced, plus a survey on multimodal embodied agents.
More from Embodied
- Developer's New Robotics BLDC Controller PCB Batch Arrives, Running Unit Tests — wavefnx · 2026-11-16
- YC-backed Ornadyne's flapping-wing drone was classified as a bird by Air Force detectors — import_jmr · 2026-11-16
- Uber starts offering Amazon's Zoox robotaxi rides in Las Vegas — Polymarket · 2026-11-16
- Ant Lingbo Embodied AI Challenge crowns TARS team after 48-hour real-robot hackathon — 新智元 · 2026-11-16
- Tesla FSD fleet tops 7 billion city miles, adding ~500M in under four weeks — XFreeze · 2026-11-16
- Ropedia Releases Xperience 2: 1,500+ Scanned Real-World Scenes Paired With Egocentric Demos — liuziwei7 · 2026-11-16