Robots Acquire Manipulation Skills in Seconds from a Single Human Video
Guangyan Chen, Meiling Wang, Te Cui, Zichen Zhou, Qi Shao, Shalfun Li, Hang Su, Roy Gan, Hao Wang, Mengyin Fu, Yi Yang, Yufeng Yue
cs.RO
2026-07-22
HOST lets a bimanual robot learn a new skill from one human video in 29 seconds, hitting 62% success, beating a 50-demo fine-tuned baseline and retaining old skills.
Teaching a robot a new skill today is expensive, slow, and self-defeating. You collect demonstrations through teleoperation (burning expert time and scarce robot hours), fine-tune offline for hours, and the parameter update overwrites shared weights, eroding skills already learned (catastrophic forgetting). Teach one new motion, forget one old one. This paper asks whether a robot can watch a single video of a human performing a task, acquire the skill at inference time in seconds, and keep what it already knows.
The hard part is the mismatch between human and robot. The video shows human hands, a particular viewpoint, a particular speed; the robot has a different body, a different camera, a different pace. This structural mismatch has three layers: temporal asynchrony (video and robot trajectory unfold at different speeds), state discrepancy (different embodiment, viewpoint, appearance), and scarce paired supervision (human-robot pairs for the same behavior are rare).
HOST's core is to turn the video from a passive conditioning signal into the active driver of the robot's prediction. It does two things.
First, it couples prediction targets to the demonstration. Using Smooth Dynamic Time Warping, it aligns the robot trajectory and the video onto a shared task progress manifold: frames are encoded with Qwen3-VL-Embedding-8B into vectors, a similarity matrix is computed, and forward-backward dynamic programming recovers a monotonic soft match. Training is self-supervised with temporal cycle-consistency loss and DTW loss. Each prediction target is then redefined to align with the video's future progression.
Second, a self-grounded prediction cascade inside a single autoregressive diffusion model runs as a causal chain of three stages: progress localization (figure out where the robot currently stands in the demonstration), future observation translation (render the upcoming video segment as the robot's own future observations, adapted to its body and scene), and action derivation (extract the action from those predicted observations).
The architecture is a dual-expert Mixture-of-Transformer on a Wan video diffusion backbone: a video expert (localization and observation prediction) plus an action expert (reduced hidden dimension, for action generation). Training is two-stage: pretrain on 193,462 same-embodiment robot-robot pairs across 229 tasks, then adapt with 5,847 human-robot pairs.
50 novel manipulation tasks, 20 trials each with randomized initial conditions:
| Metric | HOST | Comparison |
| Time per skill | 29 s | fine-tune 4.0 h (507x faster) |
| Novel-task success | 62% | +45 pts over zero-shot |
| vs 50-demo fine-tune | 62% | Wall-OSS+SFT 56% |
| Demos needed | 1 video | 50 (50x fewer) |
From a single human video, HOST reaches 62% success, beating the zero-shot baselines (pi0.5, Wall-OSS) by 45 points and beating the strongest fine-tuned baseline with 50 robot demos (Wall-OSS+SFT at 56%). It is 507 times faster than pi0.5+SFT (4.0 hours versus 29 seconds).
The bigger point is retention. On previously mastered tasks, HOST keeps its original performance while fine-tuned methods collapse: pi0.5+SFT retains only 20% of original performance, Wall-OSS+SFT retains 43% (the best among fine-tuned methods), and HOST beats that best retention by 56 points. Robustness holds: lighting variation costs 1 point, out-of-distribution objects 4, scene replacement 6, and human disturbance during execution 9, all acceptable.
For robot-learning practitioners, this paper pulls "teach a new skill" from hours of offline training down to seconds of inference-time acquisition, without sacrificing old skills. One human video expands the library; teleoperation collection and catastrophic forgetting stop being the bottleneck. The approach is also worth borrowing: rather than translating video straight to action, HOST first aligns the video onto the abstract axis of "task progress" and has the robot predict its own future along that axis, sidestepping the dead end of human-robot embodiment difference.
The authors concede three points: evaluation is on a single bimanual platform, and substantially different embodiments are untested; video carries no contact forces, so tactile information needed for fine-grained manipulation is lost (they suggest wearable sensors like tactile gloves); and as the demonstration memory grows, similarity-based retrieval may fail to separate tasks whose instructions and scenes differ only subtly. A further concern: 50 tasks is respectable for this line of work but all sit in the narrow class of bimanual tabletop manipulation, so whether "29 seconds to learn" holds on longer-horizon, more complex tasks is not shown.