World Action Modeling with Progressive Visual Planning
Fei Zhang, Zhaochong An, Duncan Frost, Yikai Wang, Pengfei Liu, Ya Zhang, Michal Drozdzal, Amir Bar
cs.AI, cs.CV, cs.RO
2026-10-02
ProWAM caches sparse progress-indexed sub-goals from one video pass, then denoises actions: 85.8% LIBERO-Plus, 75.7% randomized RoboTwin, 70% zero-shot on a real robot.
World action models (WAMs) turn a video generator's dynamics prior into robot actions. Full-imagination systems roll out a dense future video and then condition the policy on it. The near-term picture is detailed, and the cost is repeated denoising through a large video backbone, which long-horizon closed-loop control cannot afford. Zero-imagination systems ignore predicted frames at inference time. ImageWAM conditions actions on a single frame at the end of the action horizon.
That endpoint shows where the task stops. It leaves the path from the current view unspecified. Demonstrations also vary in length, so an absolute frame index breaks once execution moves to a new stage. The missing signal is progress measured from the current observation.
ProWAM replaces one endpoint with an ordered set of sparse visual sub-goals, indexed by relative progress r. The current observation is 0 and the end of the remaining demonstration is 1. Each sub-goal is the nearest frame at a chosen fraction between those two points, encoded by a pretrained VAE into a latent slot. Every replan sets the latest observation back to r=0, so the plan slides forward with execution.
Slots are appended to the video latents and share the video weights and position encodings. A sinusoidal embedding of r passes through an MLP and becomes an adaptive LayerNorm (adaLN) offset, added to the flow-matching timestep modulation. Flow matching trains a velocity field that carries noise toward data. During training the sub-goal count k is drawn uniformly from 1 to 4. At inference, k and the progress schedule can change with no architecture change.
The backbone is Wan2.2-TI2V-5B. A video expert and a lightweight action expert share attention heads in a mixture-of-transformers (MoT), with a mask on who can read whom. Sub-goal slots attend to each other and to the current frame. Action tokens attend only to the current frame and the sub-goals. The video expert never receives actions, so the video pathway can be pretrained on action-free video. The action condition is the current frame plus sub-goal features, not a dense future rollout.
Stage one trains only the video expert on large-scale robot and egocentric video, covering short-term dynamics and progress-conditioned sub-goals. Stage two jointly finetunes on robot demonstrations. Action chunks use their own flow-matching timestep, sub-goals share the video timestep, and the action loss reads sub-goal features. Each replan runs the video expert once, caches sub-goal keys and values, and reuses that cache for every action-denoising step. Video-expert forwards drop from T to 1.
In-domain LIBERO is near saturation. ProWAM scores 98.8%. ProWAM-Base, without video pretraining, and Being-H0.7 both score 99.2%. Zero-shot LIBERO-Plus opens a gap: ProWAM reaches 85.8%, above StarVLA at 83.1% and Being-H0.7 at 80.5%, with Base at 80.3%. Video pretraining is worth 5.5 points there.
RoboTwin is finetuned on clean demonstrations and tested with scene randomization. ProWAM scores 85.8% clean, close to ImageWAM at 85.0%. Randomized success is 75.7%, against 55.7% for MV-WAM, a gap of 20.0 points or 35.9% relative. On that same randomized column ImageWAM scores 37.6%, and Fast-WAM falls from 71.9% clean to 6.3%. Removing video pretraining drops ProWAM's randomized score from 75.7% to 67.4%, while the clean score moves from 85.8% to 87.6%. Clean scenes do not improve. The randomized split gains 8.3 points.
| Setting | ProWAM | Comparison | Score |
| LIBERO-Plus | 85.8% | Being-H0.7 | 80.5% |
| RoboTwin randomized | 75.7% | MV-WAM | 55.7% |
| RoboCasa365 overall | 48.1% | Paimon-0 | 58.1% |
| Real robot, zero-shot | 70.0% | DreamZero 14B | 55.0% |
| Inference compute | 18.7 TFLOPs | Joint denoising | 115.5 TFLOPs |
On RoboCasa365, overall success is 48.1%, 4th among the public methods in Table 3 and 10.0 points behind Paimon-0 at 58.1%. Atomic-Seen is 76.0%, close to the leaders at 78.7% to 80.2%. Composite-Seen is 46.5%, behind the leaders at 55.8% to 57.4%. Composite-Unseen is 18.2%, behind Phasor-m7 at 31.5%, Xiaomi-Robotics-1 at 32.1%, and Paimon-0 at 36.7%, and ahead of ABot-M0.6 at 7.9%. The abstract and Table 3 both say 4th. The introduction and Section 4.2 call the same entry 2nd overall and 2nd on unseen compositions.
The real-robot setup trains on DROID and deploys zero-shot on a Franka, with no target-scene demonstrations. The four tasks cover pick-and-place, multi-object assembly, contact-rich placement, and spatially constrained placement. ProWAM at 5B averages 70.0%, against 50.0% for π0.5 at 3B and 55.0% for DreamZero at 14B, a 15.0 point gap. Spatial placement is reported at 100%. Per-task baseline scores for that split are not in the main text.
On one H100, with action chunks of length 16, joint video-action denoising costs 115.5 TFLOPs and 581 ms. Sub-goal caching costs 18.7 TFLOPs and 313 ms: 84% less compute, 46% less latency, and 3.3 points higher randomized success in that comparison. Raising k from 1 to 4 adds 3.5 TFLOPs. k=4 peaks at 75.7% randomized. k=1 already reaches 73.8% at 15.2 TFLOPs, against Fast-WAM at 6.3% and 13.2 TFLOPs. Nearer schedules win: r in [0.1, 0.5] scores 75.7%, [0.3, 0.7] scores 70.2%, and [0.5, 0.9] scores 68.4%, with in-domain success around 86% for all three. The 7.3 point gap from far to near is larger than the 1.9 points from one sub-goal to four.
The 5B video backbone runs once per replan, so inference cost sits with cached WAMs. Action-free video carries the visual plan. Demonstrations are spent attaching that plan to actions.
One nearby sub-goal already scores 73.8%. Four sub-goals add 1.9 points. The progress band matters more: near targets beat far targets by 7.3 points. Visual randomization and novel real scenes show double-digit absolute gains. Composite-Unseen stays at 18.2%. Long-horizon composition does not move with the visual gains.
In failed cube stacking, the sub-goal blurs the two cubes together, the gripper opens early, and the cube lands off center. Texture error is still executable. Wrong contact geometry becomes a bad release. Sub-goals come from a single forward pass and are blurry. Pixel-perfect reconstruction is not the target. The policy conditions on these features with no separate geometry check.
The 313 ms figure is from an H100. Robot control rate and the number of trials per task are not reported. The 70% score is a four-task average. Spatial placement at 100% has no per-task baseline, so the real-robot gain cannot be traced to a task type.
Table 3 ranks ProWAM 4th both overall and on Composite-Unseen. The gap from 18.2% to third place at 31.5% is still wide. The introduction's average absolute gain of 11.4 points over prior WAMs does not list the terms being averaged. The margins that subtract directly from the main tables are 5.3 points on LIBERO-Plus over Being-H0.7, and 20.0 points on randomized RoboTwin over MV-WAM. The scale of action-free pretraining, and how trajectory endpoints are defined in that video, are not stated in the main text.