Masked Visual Actions turns 15 hours of robot video into a zero-shot world model
jbhuang0604 · x · 2026-07-24
Masked Visual Actions turns video models into robot world models
Researchers introduce Masked Visual Actions (MVA), a way to repurpose pre-trained video models as robot world models. The core idea is to express actions in the language of vision, letting the model answer counterfactual questions such as: if an object moves in a certain way in the clip, how will the scene evolve?
The method connects two robot-relevant capabilities:
- Forward dynamics: predict future observations from observations and actions
- Inverse dynamics: infer actions from observation sequences
According to the authors, the approach generalizes zero-shot to unseen embodiments and scenes despite being fine-tuned on only 15 hours of single-robot data. They also say the same representation works whether the “thing” is a robot or an object, which makes it useful for model-based planning, policy evaluation, and action inference.
Related event: Zero-Shot Robot World Model Trained with 15 Hours of Video(3 posts)→
More from Embodied
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Photos from ROSCon UK: BBC's robotic whale, da Vinci surgical robot, PR2 and Baxter — kamathsblog · 2026-10-27
- Agility's Digit robot to expand into two more GXO US warehouses from early 2027 — jonstephens85 · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Wayve CEO argues end-to-end deep learning should power driving, safety and world models — alexgkendall · 2026-10-27
- Mundo turns a few room photos into an explorable, simulatable 3D world — at 1.5 FPS — mark_k · 2026-10-27