The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction
Xunzhe Zhou, Yiyang Cai, Fengyi Wang, Ran Ju, Hanxiang Ren, Ruizhe Liu, Yu Zhang, Qian Luo, Feng Chen, Pei Zhou, Yi Ma, Yanchao Yang
cs.RO, cs.AI
2026-08-23
HKU grades robot video imitation by scene mismatch. Nine policies hold near 0.4 success through L2, then drop to 0.29 at L3; all stay under 13% zero-shot on unseen tasks.
People who watch someone stir soup do not store a joint trajectory. They pick up a goal, then reuse whatever is on the table. Robot learning from human video still mostly remaps trajectories: extract hand keypoints or optical flow, retarget them to an arm, replay in a near-identical scene. Move the pot, swap the spoon, and the copied motion dies even though the task did not.
Robustness suites such as COLOSSEUM, RoboTwin 2.0, and REALM perturb scenes the policy already trained on. Video-conditioned controllers like Vid2Robot are mostly tested in-distribution. HKU (with TranscEngram, Fudan, and Zhejiang University) asks a different question: can a policy watch a human clip and reproduce the intent in its own scene, even when objects change place, shape, or function. That is the bar for treating the internet's human video as robot supervision.
The Imitator Game grades mismatch between the demonstrator scene and the robot scene:
The policy sees only the video V and current observation. It is never told the level. Language-conditioned VLAs get a frozen captioner (human labels plus Qwen3-VL: intent, layout, per-arm subtasks). Video-conditioned models see the clip.
Three artifacts sit under that protocol. IG-10K holds 20,000+ paired episodes (11.7k real, 10k sim), 50+ base tasks, 200+ variants, and six domains. Real data is VR teleop on a dual-arm Realman; sim is a dual-arm Franka in ManiSkill3. Each episode ships multi-view video, 3D MANO hands, masks, and three-level language. One substitution pattern per task-level pair keeps supervision equal. Imitator Arena scores simulated SR / Sub-SR from goal predicates, and runs blind A/B with 10 volunteer judges (15,000 sim pairs, 5,000 real pairs). Nine models and fifteen trained variants cover three interfaces: VLAs (OpenVLA, RDT-1B, GR00T-N1.6, π0.5), skill retrieval (XSkill, UniSkill), and visuomotor heads (ACT / Diffusion Policy / VQ-BeT with DINOv2, SigLIP2, or VideoMAE). Backbones stay frozen; only the action head trains.
Pretraining corpora nest 15 / 30 / 45 tasks at 50 demos each. Evaluation uses five hard seen tasks and five held-out atomic skills, all four levels. Unseen tasks run zero-shot, 10-demo from scratch, and pretrain-then-10-demo fine-tune.
On simulated seen tasks, video pipelines lead.
| Method | Seen SR | Seen Sub-SR | Zero-shot SR | 10-demo scratch SR | Pretrain+FT SR |
| ACT/DINOv2 | 0.81 | 0.93 | 0.02 | 0.76 | 0.84 |
| XSkill | 0.79 | 0.91 | 0.10 | 0.35 | 0.73 |
| π0.5 | 0.73 | 0.89 | 0.09 | 0.80 | 0.85 |
| OpenVLA | 0.29 | 0.60 | 0.06 | 0.14 | 0.20 |
Zero-shot tops out at 0.13 (VQ-BeT/DINOv2). Pretraining mainly buys few-shot adaptation: 12 of 15 sim variants beat from-scratch after P+FT, and 14 of 15 rise when the corpus grows from 15 to 45 tasks. GR00T gains +0.46 SR; RDT-1B and VQ-BeT/DINOv2 lose.
On real hardware, XSkill is the most consistent: seen SR 0.63, zero-shot 0.29, P+FT 0.49, with A/B win rates 0.89 (seen) and 0.95 (P+FT). π0.5 reaches 0.51 on seen tasks and 0.04 zero-shot, and extra paired data does not lift it. The three video-conditioned models do pick up some zero-shot as the corpus grows (XSkill 0.24 to 0.33).
The hierarchy breaks at L3. Real P+FT averages:
| Level | Mean SR | Mean imitation score Q (0–10) |
| L0 | 0.42 | 6.26 |
| L1 | 0.42 | 6.22 |
| L2 | 0.39 | 6.17 |
| L3 | 0.29 | 5.62 |
XSkill holds 0.53–0.57 through L2, then 0.29 at L3. Layout change and same-category swap are tractable; using a different object in a different way is not. Simulation actually dips at L2, because the sim L2 condition also swaps which arm acts, and because finite assets plus planned motion let an L3 substitute often be handled with a trained pick-and-place habit. Hardware does not offer that shortcut.
The paper turns "learn from human video" into a ladder you can fail on. If internet video is going to supervise robots, the missing piece is functional substitution, not spatial jitter. Numbers that look healthy at L0–L2 will hide an intent interface that is not there.
Scaling the corpus or the action head will not produce intent-level imitation as a side effect. The demonstration has to remain evidence of purpose: affordance-aware grounding, explicit goal inference, representations that survive object swap. Dataset, Arena, and protocol are public.
Paired human-robot data helps few-shot fine-tuning and barely helps zero-shot. Watching the video beats reading a caption if the goal is transfer. Frozen backbones keep the comparison fair across families; they are not a claim about the best training recipe.
The dataset is hand-designed and finite. One substitution per task-level pair buys comparability and throws away variance over substitutions. The hierarchy varies scene mismatch while holding the human-robot embodiment gap roughly fixed. Simulated L3 is biased by the asset pool and scripted motion. All nine policies were adapted into this interface, not built for intent imitation.
A few caveats sit outside that list. Hardware uses 5 trials per task-level pair (10 in sim), so the 0.13 L3 drop has sampling noise. Each Arena screen has one judge; reliability comes from resampling, and inter-annotator agreement is not reported. VLA captions are dense and human-checked, kinder than auto-captions at deploy time, so language models may be slightly inflated. Frozen backbones mean visual features never adapt to the paired data, so the intent gap cannot be split into wrong interface versus untrained representation.
Failure cases also cluster at contact: thin spoons slip, waste-discard needs bimanual timing, mug hanging misses height. How much of the L3 drop is missing intent, and how much is grasp calibration after the object changes, is not separated.