Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, Yinghao Xu
cs.RO, cs.CV
2026-08-27
Zero-WAM executes unseen manipulation from in-context human videos, reaching 46.95% on seven held-out RoboTwin tasks, 29.5 points above LingBot-VA.
Zero-shot cross-task robot manipulation is stuck on how to specify the new task. Language underspecifies spatial constraints, intermediate states, and timing. VLAs stuff the task into text and still fail on unseen skills. A human demo video shows how the scene should evolve, but paired human-robot data is expensive, and the policy learns a shortcut: on seen tasks, robot history plus text already predict the next action chunk, so the human video is ignored exactly when it is needed.
Zero-WAM is a causal video-action model. One policy accepts language or a human video as the task condition and autoregressively predicts the next robot video chunk and its aligned action chunk. The video branch forecasts the world; the action branch is inverse dynamics. They share attention under a Mixture-of-Transformers with separate parameters. The backbone is Wan-2.2-TI2V-5B converted to a causal policy. Human-video latents get a height-axis RoPE shift so they do not collide with multi-view robot latents.
Data comes in two stacks. Task-diverse VA re-cuts public sets such as AgiBot and OXE by action×object, sampling 6,000-plus tasks and about 400K trajectories per epoch so repeated teleop of a few skills cannot dominate. HumanGen turns those robot trajectories into semantically matched human videos: a VLM extracts object-state changes, an image editor rewrites the first frame, Wan 2.7 or Kling 3.0 synthesizes the human clip, and a VLM scores semantics and physics. The result is 74.2K human-robot ICL pairs across 8.6K tasks and more than 45 embodiments, including sim and real subsets.
Standard next-chunk prediction is too easy on seen tasks. In-context future chunk prediction (IFP) predicts K=4 future robot video chunks at stride 2 from the current robot-video representation, without attending to the human video, so the main branch must store the task evolution in that representation. IFP heads are dropped at inference. Pre-training takes 15,360 GPU hours with a 1:5 mix of Task-diverse VA and HumanGen.
RoboTwin 2.0 holds out seven unseen tasks, 100 closed-loop trials each, three seeds.
| Method | Seven-task average success |
| WAN-Action (43 seen tasks only) | 10.98% |
| LingBot-VA | 17.45% |
| Zero-WAM | 46.95% |
Zero-WAM leads on every task. Place empty cup reaches 84.87%. Stack blocks three stays hard, but Zero-WAM is the only main-table method above zero (9.00%). Adding human videos on the 43 seen tasks without large-scale pre-training already yields 36.36%. Removing IFP drops the average to 28.55%. Masking the human video and keeping only task-balanced robot data still scores 39.44% as a text-only variant, so the resampling itself beats LingBot-VA’s raw mix.
On a real bimanual Franka, with human video and no language: object-to-container 53.3% versus language-conditioned LingBot-VA at 43.3%; three-object sequential 33.3% versus 10.0%; two-table-leg insertion 16.7% versus 0.
Cross-task generalization becomes “give a human clip at deploy time,” with no extra robot demos and no weight update. HumanGen shows that human-robot pairs can be synthesized from robot trajectories, stretching task coverage from tens or hundreds in hand-collected sets to 8.6K. IFP is a reusable anti-shortcut: the auxiliary head supervises the main representation and is discarded later. The path is already usable on tabletops. It is not yet a warehouse where you point at anything.
The discussion stays on stationary tabletop work; mobile manipulation and much longer horizons are left open. HumanGen videos are generated: they pass semantic and physics filters and can still be wrong on contact and timing. Real-world ICL is only 252 pairs. The real-robot baseline is language-conditioned LingBot-VA, not the same human-video interface. Stack blocks three is still 9%; long-horizon composition is unsolved. Pre-training costs 15,360 GPU hours. Success criteria for stamp seal and move stapler were tweaked; the authors apply the same rule to every method, but it is still a eval detail.