MaskedVisualActions turns robot motion into pixels and lifts RoboCasa success by up to 26 points
机器之心 · wechat · 2026-07-23
A Stanford, Maryland, and Harvard team proposes MaskedVisualActions, a way to render robot actions as pixel-level motion trails so a video world model can reason about them directly.
What it does
- Converts robot candidate actions into visual trajectories instead of numeric control states.
- Lets the same model do forward prediction (action future video) and inverse generation (desired object motion possible robot behavior).
- Supports different robot embodiments with the same visual interface.
Results
- Fine-tuned with about 15 hours of robot interaction data.
- In RoboCasa, video-model-assisted planning improved success rates by 7 to 26 percentage points.
- Predicted strategy performance correlated with real simulation at 0.982.
- The inverse-action pipeline reached 90% success on CoffeeServeMug.
Caveats
The model is still limited by the underlying video model: it can be optimistic, produce physically inconsistent futures, and needs extra inverse-dynamics machinery for execution.
Related event: MaskedVisualActions Turns Robot Actions into Pixel Trajectories(2 posts)→
More from Embodied
- CoRL paper: flow-matching VLAs can run 1-step Euler inference with no retraining — kaixhin · 2026-10-26
- Robotic dexterity should be judged on speed, force control and tool use, not copying humans — DJiafei · 2026-10-26
- Shenzhen company combines humanoid robots with industrial robot arms — RoboBalaji · 2026-10-26
- Realbotix, a sex-doll maker, unveils Echo humanoid with lifelike face — CyberRobooo · 2026-10-26
- Track bolt robot tightens railway fasteners in 6–7 seconds with sub-mm precision — RoboBalaji · 2026-10-26
- Tesla FSD (Supervised) Approved in Latvia, Its 10th European Country — XFreeze · 2026-10-26