AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
Jiadi You, Qize Yu, Yue Chen, Minghong Cai, Zhide Zhong, Yuran Wang, Bowen Ping, Jiaqi Liang, Zhenhao Shen, Haodong Yan, Yinchuan Li, Ruihai Wu, Xiaojuan Qi, Yingcong Chen
cs.RO, cs.AI, cs.CV
2026-09-16
AffordanceWAM predicts scalar affordance plus heatmaps, letting unlabeled human video train control. Fixed robot data: RoboCasa 63.7% to 69.4%; real tasks 42.7% to 74.7%.
Teleoperated robot video teaches control directly, but it is expensive and the scenes stay narrow. Egocentric human video is cheaper and covers more motions, yet it carries no robot actions, and the body, look, and camera do not match. Retargeting a human trajectory into a robot action pays off only when viewpoint, speed, and behavior already agree. When they do not, extra human data pulls success down.
A world-action model (WAM: one network that predicts future frames and actions together) splits that supervision. Human clips teach how the scene changes. Robot trajectories teach how to move. Most WAMs still cast the future as RGB or visual latents, so background jitter and camera motion fill the prediction, and the object region that must be touched stays implicit.
Affordance is what interactions an object allows. It does not name a hand or a joint. AffordanceWAM writes it into the future world as two streams. Scalar Affordance is a per-frame field in [0, 1] marking the task object and the region where contact is feasible. Human and robot clips share that meaning. The Affordance Heatmap paints the field back onto the RGB frame with a fixed render, so the region stays stuck to the object. Future RGB is the third stream. A World Expert, started from a pretrained video diffusion Transformer, generates all three.
The Action Expert has separate weights and the same depth. Masked joint self-attention couples the two at every layer. RGB and scalar-affordance tokens see each other. Heatmap tokens see RGB, the scalar field, and themselves. Action tokens see only RGB and the scalar field. Actions cannot write into the world prediction, so control signals do not corrupt scene dynamics. Heatmap features never reach the action expert. A heatmap is still mostly RGB appearance, and an open path would be a shortcut around the scalar field. The action expert can still read RGB. Nothing forces it to ignore appearance.
Training is flow matching. A clean target and Gaussian noise mix along a straight path in flow time τ. The network predicts velocity, noise minus data, and integration runs from τ=1 back to τ=0. The three world streams share one flow time. Actions draw their own noise and time. Human clips update only the World Expert: velocity losses on RGB, scalar affordance, and the heatmap, plus a frame-level reconstruction MSE for the scalar-field codec. Robot trajectories also update the Action Expert. Human samples get no action tokens, no pseudo-actions, and no cross-body retargeting.
Stage I warms up the world, then eases in the action loss. Action training mixes two paths, denoising against noisy world features and refining on a finished world, and the schedule shifts toward the joint path. Stage II freezes the world model and trains only the Action Expert on target-robot vision-language-action demos that have no affordance labels. Gradients stop at the world interface. Inference follows the same split: four streams generate together from noise, then a short action-only denoise runs with the predicted world held fixed. The robot executes the refined chunk.
On RoboCasa, AffordanceWAM reaches 69.4% average success with 300 downstream demos per task. At that same count, π0 scores 62.5% and GR00T-N1.5 scores 64.1%. Cosmos Policy hits 67.1% with 50 demos, and UWM hits 60.8% with 1,000. The headline table does not match demo budgets. On CALVIN ABC→D, trained in A, B, and C and tested in unseen D, average chain length is 4.22 against 4.01 for VPP, the strongest listed baseline. Rates for completing at least 1 through 5 tasks are 96.1%, 92.8%, 85.5%, 77.8%, and 69.5%, against VPP at 95.3%, 88.2%, 80.3%, 72.9%, and 64.5%.
The ablation keeps robot supervision fixed and toggles human video and affordance. The first four cells each use three seeds.
| Setting | RoboCasa SR | CALVIN length |
| Robot only | 57.3% | 3.75 |
| Add human video | 55.1% | 3.68 |
| Add affordance | 63.7% | 3.91 |
| Both | 69.4% | 4.22 |
| Heatmap path open | 63.5% | 3.85 |
Without affordance, those human videos drop RoboCasa from 57.3% to 55.1% and CALVIN length from 3.75 to 3.68. With affordance, the same videos raise 63.7% to 69.4% and 3.91 to 4.22. Human clips never update the Action Expert, so the gain has to pass through the world representation. The sign flip is an interaction of 7.9 points on RoboCasa and 0.38 length on CALVIN.
Removing only the heatmap cuts RoboCasa from 69.4% to 65.4% and CALVIN from 4.22 to 3.99. Without the scalar field, the heatmap follows appearance and the interaction structure drops out. Without the heatmap, the scalar field drifts off the object.
Scaling affordance-labeled human video from 0% to 100%, with robot exposure and update steps locked, gives RoboCasa scores of 63.7%, 63.9%, 64.3%, 65.2%, 67.1%, and 69.4%. The first 60% adds 1.5 points. The last 40% adds 4.2 points, about 74% of the gain.
On a real robot, AffordanceWAM and Cosmos Policy each get 50 fine-tuning trajectories per task and 15 trials. The macro average on five basic tasks moves from 42.7% to 74.7%: fruit-to-plate 60.0% to 86.7%, color-matched cup 40.0% to 60.0%, shape-matched duck 46.7% to 73.3%, carrot 40.0% to 86.7%, spatula 26.7% to 66.7%. Drawer closing moves from 46.7% to 80.0%. Picking at least one, two, and three fruits moves from 73.3%, 53.3%, and 26.7% to 86.7%, 73.3%, and 53.3%. Flower arrangement moves from 6.7% to 20.0%.
Human video needs no action labels and no wrist-to-arm retargeting. Affordance is a prediction target both domains share. At adaptation time the world model stays frozen, and ordinary robot demos train only the action expert. Those demos do not need affordance labels either.
With robot data held fixed, RoboCasa still moves from 63.7% to 69.4%. The extra points come from interaction coverage in the human video. The CALVIN gain over VPP is 0.21 in average length, a modest step. The real-robot gap under a matched budget of 50 trajectories is much larger, and the only control there is Cosmos Policy, at 15 trials per task.
69.4% against Cosmos Policy at 67.1% is not a matched-data win. One side uses 300 demos per task, the other 50. The fair RoboCasa neighbors are π0 and GR00T-N1.5, both at 300 demos, with leads of about 6.9 and 5.3 points. Matched-budget evidence sits mainly in the real-robot suite.
Human video enters pretraining only if it carries Scalar Affordance. The main text gives no cost for those labels. It also never checks whether a wrong field makes human data hurt again. The scaling curve is flat early: the first 60% of human data adds 1.5 points, so a mid-size labeling effort can return very little.
Opening the heatmap-to-action path drops RoboCasa from 69.4% to 63.5% and CALVIN from 4.22 to 3.85. Blocking that RGB-heavy shortcut helps. The action expert still reads RGB features directly, and the paper never measures whether control depends more on the scalar field or on appearance.
Flower arrangement at 20.0% and three-fruit pickup at 53.3% are still weak in absolute terms. Fifteen trials per task is a noisy estimate. Joint generation of three world streams plus an action refine has no reported latency. Stage II freezes the world model completely. Whether an action-only update is enough when the target scene looks more different is untested.