Agentic World Action Models
Zhuoyang Liu, Dantong Niu, Botong Zhang, Boning Shao, Jing Wang, Ruijie Zheng, Yuvan Sharma, Runyu Lu, Jiaming Liu, Renrui Zhang, Roei Herzig, Yutong Bai, Ken Goldberg, Danfei Xu, Yuke Zhu, Guanzhi Wang, Linxi "Jim" Fan, Trevor Darrell
cs.RO
2026-10-22
AW-0 jointly generates skill code, future video, and actions. Clean RoboTwin success is 80.8%, and eight real tasks average 71.1, 25.1 above the best baseline.
A world action model (WAM) generates future video together with a chunk of robot motion, so the policy can anticipate object motion and contact before it moves. A long task still needs another decision: which skill to call, which arguments to fill in, and how to order the subtasks. Predicted video and action chunks do not carry that structure.
Code-as-Policies writes the structure down as a program that selects skills, binds them to the scene, tracks progress, and replans after failure. The program sits outside the policy and can call only primitives that were already written. It has no learned model of physical change.
AW-0 puts skill code, future video, and robot actions into one generative pass. Inside the model, code is the plan. Once it passes a check, the same code can also be executed.
Three experts form a Mixture-of-Transformers, one each for code, video, and action. Each keeps its own attention projections and feed-forward layers, and they communicate through joint attention. Across experts the mask is block-causal in the order code, video, action: video and action read the code, and action also reads video.
The code expert is initialized from SDAR-1.7B. Token embeddings, layer norms, the feed-forward network, and the vocabulary head stay frozen; training covers the attention projections and modulation parameters. A frozen Qwen3-VL ViT reads the current view. Program tokens are hidden at a random ratio and the loss is computed only on hidden positions, the masked-diffusion setup. The video expert starts from Cosmos-Predict2.5-2B. Future frames go through a frozen Wan2.2 VAE, and flow matching predicts their velocity. The action expert is trained from scratch and predicts the velocity of an action chunk. Real robots use a shared 64-D state and action: 9 end-effector pose values, 22 finger joints, and 1 gripper width on each side, with unused entries zeroed. Replacing the ViT features with VAE features drops the eight-task average rubric from 71.1 to 55.6.
Autoregressive code is conditioned on its own prefixes at test time, so an early error enters the later video. Joint denoising updates all three streams over the same 4 steps. The code canvas has 128 tokens, and each step reveals the positions with the highest confidence. Video noise mixes the pretrained noise distribution with a uniform distribution in equal parts. By default the robot runs the action chunk and the code is only a plan; earlier plans go into learned memory slots. Hybrid mode checks the code at every replan. If it passes validity and progress checks, primitives run; otherwise the action chunk from that same forward pass is used.
Gemini 3.1 Pro writes programs for sampled demonstrations against primitive APIs that are declared but not implemented. Claude Opus 4.8 turns repeated motions into skills and freezes the library. Later clips reuse a cached program when contact and motion match. Pretraining uses 2,913.2 hours: 1,919.3 hours of YAM bimanual teleoperation and 993.9 hours of egocentric human manipulation, gathered into 50 skills, 23 from the robot and 27 from humans. Downstream finetuning keeps that library. Each real task has 100 demonstrations and is trained for 20 epochs.
RoboTwin 2.0 contains 50 bimanual tasks. Each task is post-trained on 50 clean demonstrations and evaluated for 50 rollouts. Clean adds no distractors. Random changes background, texture, lighting, table height, and distractors. The baselines are π0.5, EgoScale, DreamZero, and WLA. Default AW-0 shares their task demonstrations. The simulated hybrid setting does not.
| Method | Clean | Random | Average |
| π0.5 | 54.9 | 27.7 | 41.3 |
| DreamZero | 33.6 | 17.6 | 25.6 |
| EgoScale | 26.2 | 2.9 | 14.5 |
| WLA | 64.8 | 3.6 | 34.2 |
| AW-0 | 80.8 | 20.6 | 50.7 |
| AW-0 hybrid | 63.1 | 29.2 | 46.2 |
Clean success reaches 80.8%, 16.0 points above WLA, and the 50.7% average is the highest. After randomization the score is 20.6%, below π0.5 at 27.7% and above DreamZero at 17.6%. Hybrid execution scores 29.2% in the random setting and 63.1% in the clean setting. Against its own clean score, hybrid keeps 46%, while the default keeps 25%. The simulated hybrid policy was trained on a separate set of demonstrations collected by executing programs.
Eight real tasks cover two robots. YAM uses parallel grippers for arranging fruit, packing a backpack, replacing a lightbulb, and folding towels. Tianji-Sharpa uses 22-DoF dexterous hands for arranging fruit with tongs, dealing poker, flipping a scoreboard, and solving a cube. Each task has 3 seeds and 16 rollouts per seed, with randomized initial poses. The score is an additive percent rubric over substeps.
| Method | Average rubric |
| π0.5 | 43.6 |
| DreamZero | 46.0 |
| EgoScale | 35.9 |
| WLA | 39.9 |
| AW-0 | 71.1 |
The average is 71.1, 25.1 above DreamZero. Arranging fruit scores 91.7 against 70.8 for π0.5. Flipping the scoreboard scores 68.8 against 40.8 for EgoScale. Arranging fruit with tongs scores 54.2 against 22.6 for DreamZero. On lightbulb replacement DreamZero already scores 67.4, above π0.5 at 63.8, and AW-0 scores 75.0. Towel folding is close: π0.5 scores 81.7 and AW-0 scores 84.4. On that same task, hybrid execution scores 90.6 and action-chunk execution scores 84.4.
Removing the code expert drops the average to 32.9, below DreamZero. Joint denoising scores 71.1 against 54.5 for autoregressive code and leads on seven of the eight tasks. Lightbulb replacement is the exception: autoregression scores 79.2 and joint denoising 75.0. On an RTX 5090, latency is 0.19 s without code, 0.23 s for joint denoising, and 0.55 s for autoregressive code, about 2.39 times the joint-denoising latency. Removing skill functions drops poker dealing from 58.3 to 42.0. Full programs with comments beat comment-free code on all eight tasks and beat language-only subgoals on seven.
A planner left outside the policy is capped by its primitive library. AW-0 trains code together with video and action, and the default output is still an action chunk. One forward pass takes 0.23 s on an RTX 5090, 0.04 s more than the model without code.
Hybrid execution can switch between primitives and action chunks. On towel folding the switch is worth 6.2 rubric points. In randomized simulation it is the AW-0 setting that beats π0.5, and its training demonstrations are a different set.
Instruction following and skill composition carry the gains. On towel folding, AW-0 leads π0.5 by 2.7 points. After the visuals are randomized, AW-0 without the hybrid switch falls behind π0.5.
The fall from 80.8 to 20.6 is attributed to pixel-level prediction out of distribution. Simulated hybrid execution also changes the training set, so the extra 8.6 points on the random split cannot be credited to the execution switch alone. On the real robots, hybrid execution is reported only for towel folding.
The skills come from programs written by Gemini 3.1 Pro and distilled by Claude Opus 4.8. After the 50 entries are frozen, the library does not grow. Weaker annotators and tasks outside the library are not tested. The baselines keep the pretraining they already had.
The main results do not execute code. The gain in language following comes from using code as a condition during joint denoising. Continual interaction and self-improvement are stated at the end and are not tested.
The 64-D vector packs finger joints and gripper width together and zeroes the unused entries. No ablation removes the human data or the YAM data, so the remaining transfer is unknown. Real-robot rubrics and simulation success rates are not the same quantity. Eight tasks and 48 rollouts each do not show how long-horizon failures are distributed.