Latent Action as Intention Enables Efficient Future Imagination for World Action Models
Xiang Li, Yupeng Zheng, Songen Gu, Huailiang Ma, Feng Yu, Xian Nie, Shanshuai Yuan, Yujie Zang, Weize Li, Shuai Tian, Moyang Liu, Ya-Qin Zhang, Wenchao Ding
cs.RO
2026-08-26
LAWA imagines the future with latent actions, not future video. RoboCasa few-shot 65.6% and full 80.8%, beating Fast-WAM by 9.6/4.5 points at 42.9% lower latency than Joint-WAM.
A manipulation policy cannot only react to the current frame. It also needs a sense of how the interaction should unfold. World action models (WAMs) couple action learning with predictive visual dynamics. Joint-WAM denoises future frames at test time, which helps generalization, but every action chunk waits on video generation. Fast-WAM keeps video prediction in training and drops the future-video branch at inference. That is faster. In matched replications it also generalizes worse, especially with few demonstrations and under distribution shift.
LAWA, from Tsinghua, CASIA, TARS Robotics and co-authors, keeps test-time future imagination without emitting future pixels.
A compact sequence of latent actions is treated as future intention. Intention is operational: the predicted sequence of transition targets visible to the action expert. It encodes which manipulation-relevant changes should happen next, without reconstructing full future appearance.
A discrete latent-action tokenizer, close to ViPRA, is trained first. DINOv2 extracts per-frame patches. A non-causal transformer with factorized spatial-temporal attention differences adjacent frames, compresses them, and quantizes into a learned codebook. A causal forward decoder reconstructs the next frame from the previous frame plus the latent action, so the codes must carry transition information that static appearance cannot supply.
Video reconstruction is easily dominated by scene appearance, and the small contacts that decide grasp success get under-represented. A SAM-style mask decoder takes the latent action as a prompt and predicts a hand or manipulator mask. Targets come from SAM 2, so no manual drawing. Mask prediction beats optical flow in their ablation. The tokenizer is pre-trained on action-free robot and egocentric video. Motion speed is aligned per source, then sampling is rebalanced so robot clips are about 20% of expected samples.
During policy training the tokenizer is frozen and its codebook embeddings become latent-action targets. Video, latent-action, and action experts share joint attention under a structured mask: action tokens cannot see future video, but they can see the evolving latent intention. The current observation stays clean; the three future targets are noised independently. At inference the future-video branch is dropped, and only latent actions and executable action chunks are jointly denoised with flow matching. Targets are discrete; test-time imagination is a continuous relaxation in embedding space, with no nearest-neighbor projection.
RoboCasa has 24 tabletop tasks, 50 trials each. Full data is 24,000 trajectories, 1,000 per task. Few-shot uses 10%, 100 per task.
| Method | Few-shot SR | Full SR |
| DIAL | 58.3% | 70.2% |
| Fast-WAM (reimpl.) | 56.0% | 76.3% |
| Joint-WAM (reimpl.) | 64.1% | 78.8% |
| LAWA | 65.6% | 80.8% |
That is +9.6 and +4.5 over matched Fast-WAM, and it stays with Joint-WAM. Few-shot also beats DIAL by 7.3 points. Among full-data VLAs, JoyAI-RA reports 63.2%, still below this Fast-WAM replica.
On LIBERO-Plus zero-shot, models train only on original LIBERO and are not fine-tuned on the perturbation split. LAWA’s micro average is 74.4%, against 60.0% for matched Fast-WAM, 70.4% for Joint-WAM, and 69.6% for OpenVLA-OFT. Gains over Fast-WAM are 44.3 points under camera shift and 27.5 under sensor noise. Under language perturbation LAWA scores 62.8%, below Fast-WAM’s 76.9% and Joint-WAM’s 91.8%. That axis did not follow the rest.
Gaussian noise on the latent sequence at inference drops full-data RoboCasa from 80.8% to 52.2%; temporal shuffle drops it to 56.4%. The pathway is in use. Without egocentric pre-training, LAWA sits at 59.7%/76.3% and trails Joint-WAM by 3.4 and 2.0 points. After pre-training it reaches 65.6%/80.8%. Under matched video, LAWA gains 5.9 and 4.5 points from that pre-training, versus 1.5/1.7 for Fast-WAM and 1.0/0.5 for Joint-WAM.
On one A800, latency per action chunk is 196.5 ms for Fast-WAM, 338.5 ms for LAWA, and 593.1 ms for Joint-WAM, 42.9% below Joint. On four real tasks (gear, battery, block, laboratory), LAWA at 25% of demos averages 40.0%, above Fast-WAM’s 33.8% with the full set. On the two long-horizon tasks, Fast-WAM records zero successes at 25% data; LAWA reaches 45.0% and 30.0%.
The efficiency argument around WAMs is often framed as a binary: generate future video at test time, or delete imagination entirely. LAWA is a third path that compresses imagination into latent actions. Latency lands between the two baselines. Success tracks the expensive one, with a wider gap in few-shot and long-horizon real tasks.
For anyone running on a robot, 338.5 ms versus 593.1 ms is noticeable. The catch is the tokenizer: mask supervision plus action-free video pre-training. Without that, latent actions do not beat explicit future video. Code and weights are listed as to be released.
Without egocentric pre-training, the latent-action interface still loses to Joint-WAM. Language perturbation is clearly weaker, and the paper does not isolate why. Fast-WAM remains faster. Matched runs share data splits and optimization, not parameter counts or inference cost, so the system comparison is not an equal-parameter one. The mask loss is described as a manipulation-oriented inductive bias, not direct proof of learned semantics. Real-world evaluation uses 20 trials per task, a tight sample.