In-Context Robot Learning with VLM Agents
Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, Tong Wu
cs.CV, cs.RO
2026-09-17
GPT-Policy wires a frozen commercial VLM to a real robot. Human video lifts towel and notebook pickup from 0/3 to 2/3; contact tasks still need aligned action labels.
No demonstration set covers the layouts, objects, and interactions a robot will meet at deployment. Dedicated embodied models such as GEN-1.5, S1, and Zero-WAM already treat one-shot video as a task spec. They were trained as robot policies.
Off-the-shelf vision-language models have not been measured the same way. The question is whether a frozen commercial VLM (GPT-6 Astra, Claude Fable, Kimi K3) can turn demonstrations, goal images, and interaction history into executable, checkable robot commands, with no gradient step and no task-specific parameters left behind.
GPT-Policy makes that question comparable. One tool interface and one success criterion, then swap the context condition or the model.
Three pieces sit in a closed loop.
Policy weights never move. Adapters only handle frames, IK backends, and vendor message formats across YAM, ARX X5, and Morphi Kino. The Cartesian planner does not check collisions. Bimanual segments are synced to the slower arm.
Each task is three trials. Success is geometric and semantic at the final scene. The main model is GPT-6 Astra.
| Task | No context | Human video | Robot video | Video+action |
| Pick red towel | 0/3 | 2/3 | n/a | n/a |
| Pick up notebook | 0/3 | 2/3 | n/a | n/a |
| Unscrew bottle cap | 0/3 | n/a | 2/3 | 3/3 |
| Unplug and reseat | 0/3 | n/a | 0/3 | 2/3 |
Human video carries no robot action labels, and average cost falls with success: towel decisions 96.3 to 76.7 and time 24.6 to 18.9 min; notebook 94.0 to 66.7 decisions and 24.6 to 16.1 min.
Contact-rich tasks need more than pictures. Plug reseating stays at 0/3 with robot video and reaches 2/3 only after aligned action references. Bottle opening keeps 205 action samples against 13 video keyframes, so motion between frames is no longer guessed.
Goal images, self-interaction history, and live human-robot interaction each hit 3/3. Those rows have no matched no-context control. On a single towel run, human video moved GPT-6 Astra from 55% task progress to 100%, 24.63 to 15.85 min, and 12.047M to 4.956M tokens. Under the same video, Fable 5.1 reached 30% progress and Kimi K3 20%. Progress is not success rate; this is not a ranking.
This is a boundary sketch, not a deployable policy. High-level procedure can transfer from a human video. Contact precision and safety do not.
The useful setting is lab-scale few-shot task specification: show a human video or a goal image and let the model get the steps right. Stable contact, two-arm coordination, or unattended operation is out of scope. The planner does not check collisions, and the discussion records repeated inter-arm hits.
Against robot-trained ICL models, the measurement is how much a general VLM already has, not how much another training round would add.
The task series is small, ablations are incomplete, and pretraining is unobserved, so causal and novel-skill claims do not stand. Three trials per cell cannot separate 0/3 from 2/3 from sampling noise. Goal-image and interaction rows lack a no-context control, so 3/3 there does not by itself show that the context helped.
Accepted references and settled joints are not task success. Model-declared completion is tracked separately from physical success. Cross-embodiment transfer and reliable autonomy are not evaluated at a scale that would support them.