World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal
Yehang Zhang, Haojian Huang, Yifan Chang, Jianchong Su, Bohan Zhou, Yingjie Xu, Wosong Chen, Tianhao Zhou, Chenxu Wang, Tianyi Zhang, Yangkai Wei, Wenqian Li, Shiyuan Deng, Yinchuan Li, Ying-Cong Chen, Zexi Li
cs.RO, cs.AI
2026-09-24
HKUST(GZ) lets a VLM rehearse actions in contact views. Skills grown only on LIBERO-90 reach 75.6% on LIBERO-Pro; Qwen3.5-9B traces lift OOD success from 1.7% to 43.3%.
General VLMs can judge space and still rarely fly a robot. Fine-tuning them into VLAs tends to wash out open-ended reasoning. Code-as-policy agents emit programs over perception and control APIs, and learn whether a pose is legal only after the program runs. Visual harnesses such as Show-Harness and VIA show the scene and let the VLM pick actions, but miss three things: the view is not centered on the current contact, an issued action cannot be taken back, and an offset seen in pixels must be rewritten as coordinates.
The question is how a general VLM can look, try, and revise with basic primitives for the whole episode.
World Action Agent leaves VLM weights alone and changes what the model sees and how a decision takes effect. At each step the main agent picks a tool: query space or skills, build or edit an action proposal, or execute. Queries and proposals do not move the robot, so rehearsal can loop.
The workspace has three parts. Contact views pick cameras from the scene cloud, robot geometry, and the current interaction, with roughly orthogonal ground projections so gripper-object-target errors can be read on two axes; a global view keeps task context. For rehearsal, the agent marks a target on the canvas, the harness runs IK and cuRobo, overlays a translucent robot, and reports feasibility. Failures get edited, or an Imagination Agent iterates the proposal in a separate context. In-view correction lets the agent drag a reference point in a contact view; calibration turns that image displacement into a bounded end-effector move.
Procedural knowledge uses the same interface. A skill stores applicability, relational steps, key states with reference images, and outcome checks; objects and magnitudes are resolved in the current scene. A Skill Agent compares reference images to the live canvas; the main agent sees only text, so old frames are not mistaken for now. The library starts from text-only seeds written by Codex (GPT-5.5), then a Learner / Editor / Reviewer loop grows it from one expert trajectory per LIBERO-90 task plus human Canvas-GUI corrections, frozen before eval. Separately, 1,774 main-agent tool calls from 112 successful episodes LoRA-tune Qwen3.5-9B; sub-agents and grounding stay on Gemini 3.7 Flash.
Backbone: Gemini 3.7 Flash. LIBERO-Pro, six splits (Object / Goal / Spatial × pose swap and task perturbation), skills grown only on LIBERO-90:
| Setting | Mean success |
| End-to-end π0.5 | 12.8% |
| Show-Harness (same backbone, no skills) | 6.7% |
| WAA zero-shot | 28.9% |
| WAA + seed skills | 43.3% |
| WAA + evolved skills | 75.6% |
| ASPIRE | 72.0% |
On the two Spatial splits WAA reaches 80.0% and 73.3%, against ASPIRE at 51.0% and 60.0% and CaP-Agent0 / RATs at 12-31%. ASPIRE still leads both Object splits and Goal Pos. Swapping the simulator cloud for fused RGB-D or VGGT drops the mean to 71.1% and 68.9%; Spatial still beats every baseline. One stove demo yields 10/10 on the matching LIBERO-Pro task.
Versus Show-Harness: about 31 model calls, 150 s, $0.20 per episode, against 120 calls, 874 s, $0.50.
On robosuite, skill-free WAA already completes every lift and stack and gets 60% restack; frozen LIBERO skills take restack to 100%. CaP-Agent0 with RATs skills averages 63.3%. After SFT, Qwen3.5-9B goes 0.0%→55.0% in-domain and 1.7%→43.3% out of domain.
The VLM is the pilot and the planner is the hands: rehearsal blocks illegal grasps, drags fix centimeter placements, and skills reuse relations rather than memorized coordinates. If you already drive a Gemini-class model for manipulation, contact views plus rehearsal are the piece worth copying, not another VLA. A 9B model can learn to fly the same harness, which is the cost lever. 75.6% versus 72.0% is a narrow win, and ASPIRE still owns Object.
The authors say the backbone bounds the system: Gemini 3.7 Flash integrates multi-view cues unevenly, and the harness cannot fix that. Tens of calls per episode remain. The 9B pilot is trained on LIBERO-Pro success traces, unlike the frozen skill library, so in-domain numbers can leak. Sub-agents stay Gemini; the 9B model never flies solo. VGGT costs 6.7 points versus sim clouds, and real calibration error will be worse. Code-as-policy is weak on open geometry, yet ASPIRE still wins object-centric splits. WAA does not sweep the board.