Transferring the Intelligence of VLMs to Robotic Control
Meng-Hao Guo, Zhe-Han Mo, Jia-Jun Wang, Yi Zhang, Kejin Wang, Yi-Xuan Deng, Jia-Peng Zhang, Yongming Rao, Shi-Min Hu
cs.RO
2026-09-19
RoboDawn hooks a frozen VLM to discrete move/rotate/gripper commands. GPT-6 Astra hits 53.2% zero-shot and 73.6% one-shot on RoboTwin 2.0 C2R, above π0.5 at 46.0%.
VLA and WAM pipelines take a pretrained VLM and train an action head on robot data. That data is expensive, embodiment-specific, and recent work shows that action fine-tuning can blunt instruction following and reasoning. Robot corpora are orders of magnitude smaller than the vision-language mix that built the VLM, so large updates overfit a narrow distribution.
RoboDawn asks a narrower question: can the intelligence already sitting in a frozen VLM be transferred to physical control through a human-readable action interface and a handful of in-context demos, with no parameter updates.
The interface looks like a game pad, not a joint-angle stream. A gripper interaction point (GIP) sits at the midpoint of the two fingertips. Translation, rotation, and grasp are all defined relative to that point, so the VLM is not asked to reconcile wrist pose with contact.
Commands are few:
Each command is expanded into a planned motion and run to a stop before the next observation. At every round the VLM sees multi-view images, robot state, last-round execution feedback, and memory, then emits a command sequence plus a short scratchpad (progress, plan, notes).
Demos come in two layers. A shared command primer shows what each primitive does to the scene. Task demos take expert trajectories, compress them to end-effector waypoints, and rewrite them as the same semantic commands the online model can issue. Simulation uses scripted experts; real robots use teleop. Long-horizon RoboDojo clips the in-context image budget at 16 frames per round, keeping grasp, rotation, and completion frames.
RoboTwin 2.0 C2R has 50 bimanual tasks. Trained baselines follow the official protocol: 50 clean demos per task, joint post-training, evaluation under domain randomization. RoboDawn never updates weights; demos enter only as context.
| Method | Data | Success |
| π0.5 | full post-train | 46.0% |
| LingBot-VLA | full post-train | 50.4% |
| HarnessVLA (Claude Code) | full post-train | 58.4% |
| RoboDawn GPT-6 Astra | zero-shot | 53.2% |
| RoboDawn GPT-6 Astra | one demo | 73.6% |
One demo beats HarnessVLA by 15.2 points. Gemini-3.8-Flash goes 47.0% zero-shot, 62.2% one-shot, 65.4% four-shot, then 62.7% at eight shots: extra context starts to hurt. Under one-shot, model class dominates: GPT-5.6-Luna 14.4%, GPT-5.6-Sol 43.2%, Seed-2.1-Pro 45.0%, Gemini-3.8-Flash 62.2%, GPT-6 Astra 73.6%.
Zero-shot ablations: drop reasoning 34.8%, drop grid localization 32.4%, drop the command primer 44.0%. Grid and reasoning matter more than the primer.
On RoboDojo (42 tasks, 5-run mean) zero-shot success is 35.67% with progress 39.92; one-shot is 47.17% / 54.63. Full-set DM0.5 sits at 19.34%. Raising the command budget from 60 to 240 lifts one-shot from 31.2% to 47.2%. Extra test-time steps buy success.
Real-world, Gemini 3.8 Flash, zero-shot: Franka block-in-basket 9/10, stacking 5/10, Piper cloth fold 0/10. Failures cluster on end-effector rotation, which the paper attributes to sparse 3D orientation examples in web pretraining.
The cost is latency. With Seed-2.1-Pro, inference is 9.74 s for 3.4 commands against 2.09 s of motion (I/M 4.65). π0.5 infers in 101 ms (I/M 0.037). The interface buys deliberation, not 100 Hz servo.
For generalist manipulation, collecting more robot data to train a VLA is no longer the only path. If the action space is rewritten as spatial primitives a VLM already understands, a closed-source multimodal model can sit as the high-level policy and leave trajectory generation to a classical controller. The jump from one demo also says interface alignment can be cheaper than another 2,500 demonstrations.
This is incremental. Fine insertion, high-rate force control, and collision safety are out of scope. Use it on tabletop tasks that can wait a few seconds, not on contact-rich 100 Hz loops.
The authors list four: slow inference, rotation harder than translation, discrete steps too coarse near contact, and unsafe collisions. Failure cases match: a coin misses the slot, IK knocks tiles during a mahjong kong, pouring stops early and the model calls done.
The comparison is not fully matched. Trained baselines see 50 clean demos per task and are tested under randomization; RoboDawn's one-shot demo also comes from the clean set. Zero-shot already beating π0.5 still holds. Real-world covers three tasks at 10 trials each. The cloth-fold zero is explained as missing orientation data, without a rotation-only ablation. GPT-6 Astra and Gemini-3.8-Flash are closed APIs, so the recipe is a harness plus a bill, not a weight dump.