H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models
Dingyi Rong, Yue Shi, Chaofan Ma, Jiezhang Cao, Zongrui Wang, Zeyu Zhang, Yao Mu, Guangtao Zhai, Ning Liu
cs.RO, cs.CV
2026-08-13
H2R-Bench tests whether video models faithfully transfer 120 human demonstrations to two robot embodiments; across 11 models, video quality barely tracks transfer success (rho=0.14).
Robot learning is starved for data, and collecting robot demonstrations is expensive and hard to scale: it needs specialized hardware, teleoperation interfaces, calibrated cameras, safety constraints, and repeated physical execution. Egocentric human manipulation video is abundant by contrast, and it naturally records object affordances, hand-object contact, manipulation intent, and physical state changes. The natural idea is to use video world models as a bridge, converting human demonstrations into robot manipulation videos to fill the robot-data gap.
But one capability has gone largely unmeasured: cross-embodiment transfer. Existing video benchmarks (VBench, WorldModelBench, RoboWM-Bench, and others) either score raw visual fidelity or evaluate videos generated from text or robot-centric states, and none evaluates faithfully converting a human demonstration into a different robot embodiment. They cannot diagnose failures unique to human-to-robot transfer, such as swapping in the wrong end effector or producing interactions that do not match the source. The hard part is not generating a video that looks robotic; it is preserving the manipulation evidence in the source while changing the execution to another embodiment.
H2R-Bench evaluates relative to the source video. Given an egocentric human manipulation video and a target embodiment, a model generates a robot video of the same task. The source video is treated as evidence of what happened, not as a visual style reference. A successful output preserves the task goal, the actions needed to reach it, and the visible interaction between actor and objects. No pose or trajectory imitation is required: a gripper and a dexterous hand may solve the task differently, as long as the new strategy stays compatible with the requested embodiment and the source task.
The benchmark uses 120 egocentric clips from the EgoDex test split, each paired with two target embodiments (parallel-jaw gripper, dexterous hand), yielding 240 cases. The sources are evenly spread across six manipulation families (20 each): rigid rearrangement, mechanism actuation, insertion and assembly, deformable configuration, bulk-material transfer, and surface or material change. Annotations are generated first by Qwen3.7-Plus (initial and final states, relevant objects, required actions, source-side contact evidence) and then manually verified case by case.
Each generated video is scored on five dimensions. M1 goal-state completion, M2 action-event completion, M3 functional contact transfer, M4 embodiment correctness (set to zero if no robot is present or a human still does the manipulation), and M5 task-agnostic video quality. The first four are scored independently by three multimodal judge models (Gemini 3.5 Flash, Qwen3.7-Plus, GPT-5.4) on a 0-4 rubric and averaged; the aggregate H2RCore weights contact and embodiment 0.30 each (60% combined), goal and action 0.15 each, and video quality only 0.10. At generation time each model uses its strongest public source-conditioning interface: video-conditioned models receive the full clip, image-conditioned models receive ordered source frames, and the main setting supplies no target-robot reference image, specifying the embodiment in text alone.
Eleven models are tested: five proprietary (Seedance 2.0, Wan2.7, Kling-V3, Veo 3.1, Grok Imagine Video) and six open-source (Wan2.2, LTX-2.3, HunyuanVideo 1.5-I2V, SkyReels-V3-R2V, LongCat, Mitty-EPIC14B).
| Rank (gripper / hand H2RCore) | Model | Score |
| 1st | Seedance 2.0 | 77.3 / 84.6 |
| 2nd | Wan2.7 | 76.5 / 83.1 |
| 3rd | Kling-V3 | 74.5 / 81.7 |
The top three are all video-conditioned models. Goal and action scores are similar across the leaders; the separation comes mostly from contact transfer and embodiment correctness. The failures are telling: Veo 3.1 posts the highest gripper goal-completion score (0.725) yet only 0.100 on embodiment correctness; HunyuanVideo has the best video quality on both targets (0.806, 0.808) but its contact score stalls at 0.185 and embodiment near zero. Frame-conditioned models score almost zero on embodiment (Wan2.2 and LongCat at 0.000, SkyReels at 0.004).
The sharpest conclusion: video quality barely tracks real transfer. A VBench-style quality score sits in a narrow 0.73-0.81 band, while H2RCore ranges from 30.0 to 84.6, and their rank correlation is only 0.14. The model with the best video quality (HunyuanVideo) lands near the bottom of H2RCore.
Embodiment matters too: switching to the dexterous hand raises H2RCore for 9 of 11 models, by 3.3 points on average, and contact score rises for all 11 (+0.055). The reason is that a dexterous hand is morphologically closer to the human actor in the source, so contact survives actor replacement more easily. Three human raters independently rechecked the metrics; their agreement with the automated scores reaches Spearman 0.883.
The practical value is the first credible ruler for whether video world models can serve as a source of robot data, and it is calibrated exactly where models deceive most easily: a good-looking video does not mean the task is done right or the right robot was used. For anyone working on human-to-robot video generation, H2R-Bench decomposes failure into four tiers (goal, action, contact, embodiment), so you can see which link a model breaks. The conclusion is also a cold shower: even the strongest current video models still have a clear gap on cross-embodiment manipulation transfer.
Scale is modest: 120 source videos, six families, two embodiments, narrower in coverage than general video benchmarks. The sources all come from EgoDex, skewing toward tabletop domestic scenes; industrial or mobile manipulation is untouched. The first four metrics rely on multimodal judge models; three human raters back them up with high correlation, but judge bias cannot be fully ruled out, and the M5 MUSIQ and CLIP-aesthetic components are proxy metrics, not real user perception. The benchmark measures only whether a transfer is correct; it never asks whether the resulting video actually helps train a robot policy, so that downstream step remains empty.