Modality-Autoregressive World-Action Models
Adam Hung, Bardienus P. Duisterhof, Deva Ramanan, Jeffrey Ichnowski
cs.RO
2026-09-16
ModAR autoregressively denoises future tracks, DINO, depth, then RGB before actions. On RoboTwin it reaches 75% average success versus 72% for 6B video-pretrained Flex-π, at about 1/20 the training FLOPs.
World-action models jointly predict future observations and robot actions, usually as RGB. Depth, pretrained visual features, and point tracks carry geometry, semantics, and motion more directly, but how to combine them inside a WAM is still open. Many systems also start from a video generator, so formulation, representation, and data mix are entangled.
Carnegie Mellon’s ModAR trains from scratch and isolates which future to generate, in which order, before the action head.
Given the current multimodal observation, proprioception, and a discrete task label, the model generates future RGB, depth, DINO features, and point tracks, plus the action sequence. Tracks come from CoTracker3 queries at patch centers; DINO tokens come from a frozen DINOv2. A shared diffusion transformer runs cross-modal blocks, then modality-specific experts and heads. Task embeddings and proprioception enter through adaLN.
Generation order is tracks → DINO → depth → RGB, with actions last, so the action step is an inverse-dynamics read of the imagined future. Training keeps a clean context copy and a noisy prediction copy per modality, with a block-causal mask so later targets stay hidden. Context blocks get injected noise during training to blunt cascade error; inference stays clean. The objective is JiT-style x-prediction. Actionless human video supervises available future modalities only.
Four formulations share backbone and data budget: Unified co-denoising, independent per-stream noise at train time, Disjoint futures and actions, and Action-only.
RoboTwin uses six bimanual tasks and 50 trials each. Models start from 50 action-labeled demos per task, then add actionless demos to 250 and 1,250. ModAR has the highest average success at every scale. It also gains the most from actionless data: 66% to 76% (+10 points), versus about +1 for Unified. Disjoint beats Action-only on average and then falls as actionless data grows.
Tracks, DINO, and depth add up. Extra future RGB on top of those three does not help consistently; the paper reads RGB as high-variance appearance with little extra structure. Ablations drop when context noise is removed or the order is reversed. Dropping RGB barely moves the average.
A sampling-budget objection is fair: ModAR uses 8 Euler steps per stream, 40 in total, against 8 for the others. Raising Unified, Disjoint, and Independent-noise to 40 steps does not close the gap (Unified 67% to 59%, Disjoint 55% to 54%, Independent-noise 34% to 37%, ModAR 75%). Replacing both native action heads with one separately trained inverse-dynamics model still favors ModAR’s futures.
A 30.1M from-scratch tracks–DINO–depth ModAR reaches 75% average success. Flex-π, a 6B Wan2.2-initialized WAM, reaches 72% with about 20× the training FLOPs. That is a system comparison, not a matched architecture ablation. On three real bimanual tasks (stack cups, fold towel, place in drawer), ModAR also beats Unified and Action-only and improves when human video is added.
The hard takeaway is not another world-model name. Future RGB is a weak default. Structured futures (tracks, DINO, depth), generated easy-to-hard, are what make actionless video useful. A 30M scratch model beating a 6B video-pretrained system on in-distribution tasks means “start from a big video model” is not the only path when the task set is closed and demos are controlled.
The paper also pulls formulation debates onto one backbone: joint denoising, auxiliary futures, and imagine-then-act can be measured without swapping the rest of the stack.
The task set is small and uses discrete task IDs, not language commands, so it does not speak to new objects, new scenes, or open instructions. Sequential generation adds latency. The order is chosen, not searched, and the only order ablation is reverse. The Flex-π comparison mixes size, pretraining, and implementation. Independent-noise failing from scratch is not a verdict on large video-pretrained systems that use the same trick. Success rates are point estimates over 50 or 30 rollouts with no confidence intervals.