A 15-hour video fine-tune turns masked trajectories into a robot control interface
Hadi Alzayer · hf · 2026-07-22
Masked Visual Actions proposes a pixel-space control interface for video models that is grounded in physical manipulation.
Instead of using abstract action tokens, the method encodes action as a partially revealed trajectory of an entity in video. Revealing robot motion turns the model into a forward dynamics predictor; revealing desired object motion lets it infer robot behavior that would produce that outcome. Fine-tuned on only 15 hours of masked examples from real and simulated videos, one checkpoint shows strong fidelity and controllability across scenes and embodiments. The model can evaluate imagined rollouts against real execution, rank candidate futures for planning, and synthesize robot motion from desired object motion.
Related event: MaskedVisualActions Turns Robot Actions into Pixel Trajectories(2 posts)→
More from Embodied
- Mundo turns a few room photos into an explorable, simulatable 3D world — at 1.5 FPS — mark_k · 2026-10-27
- Zurich Physical AI Hackathon puts attendees in a room with $1.5M of humanoids — ___Mufasaa · 2026-10-27
- Apple responds after iPhone Duo inner displays crack along the hinge — mobileraj · 2026-10-27
- VLA models encode task concepts as curved multi-dimensional geometry, 19-author study finds — ninamiolane · 2026-10-27
- Munro Live Posts Full Cybercab Engineering Deep-Dive Video — yunta_tsai · 2026-10-27
- Tesla's Robotaxi Fleet Keeps Growing in Austin — yunta_tsai · 2026-10-27