A 15-hour video fine-tune turns masked trajectories into a robot control interface
Hadi Alzayer · hf · 2026-07-22
Masked Visual Actions proposes a pixel-space control interface for video models that is grounded in physical manipulation.
Instead of using abstract action tokens, the method encodes action as a partially revealed trajectory of an entity in video. Revealing robot motion turns the model into a forward dynamics predictor; revealing desired object motion lets it infer robot behavior that would produce that outcome. Fine-tuned on only 15 hours of masked examples from real and simulated videos, one checkpoint shows strong fidelity and controllability across scenes and embodiments. The model can evaluate imagined rollouts against real execution, rank candidate futures for planning, and synthesize robot motion from desired object motion.
More from Embodied
- Roundup: humanoid robotics papers from last week — carlosdponx · 2026-07-22
- Disney’s Olaf robot appears at SIGGRAPH Real Time Live — carlosdponx · 2026-07-22
- Tesla puts Cybercab on display at its Tesla Diner — elonmusk · 2026-07-22
- Dyna says robotics research is only a hypothesis until it survives real-world deployment — JasonMa2020 · 2026-07-22
- Samsung is expected to unveil Galaxy Glasses at Galaxy Unpacked tomorrow — chrisgrayson · 2026-07-22
- HarnessVLA lifts LIBERO-Pro success to 82.4% by freezing the VLA and adding a planner — 机器之心 · 2026-07-22