CASTER formalizes zero-shot robot manipulation from synthetic videos as constrained optimization

mangahomanga · x · 2026-10-20

Researchers from Virginia Tech and Johns Hopkins (Yinlong Dai, Homanga Bharadhwaj, Dylan Losey) released CASTER, a paper on converting synthetic videos from generation models into real robot trajectories for zero-shot manipulation — no real human demonstration data required.

Core idea: prompt a video generator to synthesize the task in the robot's current environment, then extract object trajectories as an initial guess
Key formalization: transfer is a constrained optimization problem — refine trajectories (e.g., in a digital twin) while enforcing physical feasibility and semantic consistency
Key insight: useful video generation must probabilistically seed the search within a semantic basin of attraction, where motion can be locally refined without losing task intent
The authors show state-of-the-art approaches are instances of this formalism; paper, code, and dataset are open-sourced

Original post →

CASTER formalizes zero-shot robot manipulation from synthetic videos as constrained optimization

More from Embodied