MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation
Bingxuan Li, Siqi Song, Yizhuo Wu, Jiarui Yao, Tong Zhang, Huan Zhang
cs.RO
2026-09-30
A frozen VLM plus mid-level moves and async monitoring hits 66.7% on LIBERO-PRO base and 53.8% under perturbations, against at most 13.3% and 19.2% for prior zero-shot methods.
Vision-language-action policies map images and instructions onto low-level joint commands. On the unperturbed LIBERO-PRO suites, a fine-tuned π0.5 reaches 98.3%. Run zero-shot, the same family of policies averages 0.0% on those base suites once object appearance, layout, wording, or scene structure changes. Closing the gap usually means more robot demonstrations and another fine-tune. The policy stays tied to that robotics data, so a stronger general-purpose vision-language model never becomes the controller.
Agent systems leave the VLM upstairs as a planner and pass grounding and motion to action experts, segmenters such as SAM, skill libraries, or coding agents. Extra modules lengthen the stack. Gains in the VLM do not shorten it.
MotorMind tests a tighter loop. One frozen VLM looks at the scene, issues an action, and revises from execution feedback, without those external models.
A 240-question quiz splits local skill into three buckets of 80: choose the next action, judge whether the last step made progress, and decide whether the subgoal is done. Action selection is the worst bucket on every model tested, from 18.75% to 60.00%. Qwen3.8-Flash-Next scores 36.25%, 55.00%, and 65.00% at 0.276 seconds a query. GPT-6 Astra leads at 60.00%, 76.25%, and 82.50%, and a query takes 8.724 seconds. The answer spaces differ, so the three rates are not one difficulty scale. Qwen is still the default backbone. Proposals stay short enough to revise, and checks stay cheap enough to repeat. The quiz shows shaky local decisions. It does not show that a particular control abstraction caused the errors.
The harness follows that constraint.
The benchmark is LIBERO-PRO's Goal, Spatial, and Object suites, with Semantic, Object, Position, and Task perturbations. MotorMind uses no LIBERO demonstrations.
| Method | Setup | Base success | Perturbed success |
| MotorMind (Qwen3.8-Flash-Next) | zero-shot | 66.7% | 53.8% |
| CaP-X, best prior zero-shot | zero-shot | 13.3% | 19.2% |
| π0.5 | task fine-tuned | 98.3% | 63.3% (Figure 1) |
| OpenVLA-OFT | task fine-tuned | 98% via time score | 51.2% |
Zero-shot π0.5, MolmoAct2, OpenVLA-OFT, and GR00T N1.5 all average 0.0% on the base suites. The 51.2% perturbed rate for fine-tuned OpenVLA-OFT is stated in the text. Its base wall time is 5.7 seconds and its time score is 1033.5, which the paper's definition converts to about 98% success. No separate base average is printed for that run.
With Qwen, the base split is Goal 45.0%, Spatial 75.0%, Object 80.0%. The four perturbation rates are 58.3%, 46.7%, 51.7%, and 58.3%. Across 240 perturbed episodes, 53.8% is 2.6 points above fine-tuned OpenVLA-OFT and below the 63.3% Figure 1 gives fine-tuned π0.5.
Speed is the larger gap. MotorMind averages 223.4 seconds on base episodes, time score 17.91. Fine-tuned π0.5 takes 5.9 seconds, time score 997.1. Under perturbation MotorMind takes 248.5 seconds, time score 12.99. Time score is 60 times the success percentage, divided by mean wall-clock seconds, in percentage points per minute. It is neither a dollar cost nor a compute cost. Read it next to success and wall time. A fast failure is not a capable policy.
Four groups change the world during execution, with 10, 10, 5, and 5 tasks: dynamic reasoning 70%, scene shift 90%, conveyor manipulation 80%, prompt shift 60%. The best baseline in each group is 30%, 70%, 40%, and 60%. Prompt shift ties CaP-X.
On a real xArm6 with RealSense D455 cameras, and with no fine-tuning, direct placement succeeds 98% of the time, human mid-run object moves succeed 92%, and the pool is 95%. Four semantic instructions, five trials each, land at 80%, 100%, 100%, and 60%.
GPT-6 Sol (Medium Reasoning) lifts the base average from 66.7% to 83.3%. Spatial wall time rises from 199.0 seconds to 370.4 seconds. The per-suite writeup moves Spatial from 70% to 80%, Object from 80% to 100%, and Goal from 50% to 70%. The main table's Qwen split is 75.0%, 80.0%, and 45.0%. Both places quote 66.7%. The suite-level inputs are not the same triple.
Against that 66.7%, dropping replanning leaves 36.7% (30.0 points lower, 103.3 seconds), dropping the verifier leaves 60.0% (6.7 points lower), and dropping the planner leaves 0.0% (96.2 seconds).
If the arm can execute Cartesian mid-level commands, a new task needs no demonstrations of its own, and the stack needs no SAM, coding agent, or learned action expert. The backbone swaps in place. Qwen to GPT-6 Sol moves the base average from 66.7% to 83.3% without a new control architecture.
Inside the training distribution, a specialized VLA is still ahead. Fine-tuned π0.5 reaches 98.3% in 5.9 seconds. MotorMind reaches 66.7% in 223.4 seconds. The case for this harness is a long list of tasks and no time to collect demos: zero-shot success moves from 13.3% to 66.7%, and perturbed success sits slightly above fine-tuned OpenVLA-OFT at 51.2%. A line that cares about cycle time will not run the loop.
The remaining failures are mostly visual grounding and declaring the task finished too early. Grounding dominates semantic, object, and position shifts. Premature completion clusters on task shifts. Thin action knowledge shows up as the same useless motion proposed again and again. The paper expects stronger VLMs to shrink these three gaps. The diagnostic section is explicit that 240 local questions do not prove repeated checks will erase them.
Each adaptive group has only 5 to 10 tasks, and prompt shift does not beat CaP-X. The robot result is tabletop placement. 95% is not a real-robot LIBERO number. Figure 6, whose caption mentions a single seed, disagrees with the main table on Qwen's Goal / Spatial / Object split: 50/70/80 versus 45.0/75.0/80.0. The text never says which triple to trust. The monitor can only halt. A correction costs another VLM call. Repeated grounding errors do not get cheaper just because the check is asynchronous.