Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data
Ye Wang, Pei Lin, Xiong-Hui Chen, Haoqi Yuan, Zhixuan Liang, Yiyang Huang, Anzhe Chen, Zixing Lei, Jie Zhang, Tao Zhang, Haoyang Li, Tong Zhang, Chenxi Xiao, Ziyuan Jiao, Qin Jin
cs.RO
2026-08-04
Ego2Robot converts ~1,940 hours of egocentric manipulation video into 18,561 hours of robot training data across 15 morphologies via retargeting, robot-arm rendering, and 3-level curation; mixed 1:1 with robot data it beats robot-only pretraining on out-of-distribution generalization and real-robot tasks.
Vision-language-action (VLA) models can already drive robots through complex manipulation, but their generalization is bottlenecked by the same old problem: real teleoperation data is expensive, scarce, and monotonous. Datasets like Open X-Embodiment, DROID, and AgibotWorld are capped by hardware availability, teleoperation cost, and limited interaction diversity.
Egocentric human video is the alternative. Hand-interaction clips are abundant, span messy scenes and many objects, and carry manipulation priors robots cannot easily learn. The catch is the embodiment gap: retargeting hand motion into robot trajectories and replacing the human arm on screen with a rendered robot arm has worked at small scale and for single tasks, but nobody had scaled it to useful VLA pretraining with measurable generalization gains.
Ego2Robot converts egocentric video to robot training data in three stages.
Action alignment. From 21 hand keypoints it derives a virtual fingertip (0.7 index, 0.3 middle), then the gripper center, opening width, and a three-axis grasp frame to get an end-effector trajectory. Savitzky-Golay filtering and Gaussian-weighted SLERP smooth per-frame jitter. Because human hands move much faster than robots, frames are subsampled per source (ANT and EgoDex to 60% of their frame rate, EgoVerse to 45%, ViTRA to 25%) to match the robot speed distribution.
Visual alignment. SAM 3 segments the human arm, and ProPainter does temporally consistent inpainting to remove the arm and rebuild the background. The critical step is finding the robot base pose: egocentric video has no physical base to reference, so the authors grid-search around the trajectory centroid, scoring each candidate by inverse-kinematics feasibility at keyframes, and pick the placement with the highest reachability. This is redone for each of 15 morphologies, since different arm lengths and joint layouts need different base positions for the same trajectory. IK is then solved per frame in MuJoCo and the rendered arm is composited into the scene by depth ordering.
Quality curation runs three filters: L1 drops frames with IK failure, self-collision, or action outliers; L2 removes trajectories with discontinuities or too many invalid frames; L3 has a vision-language model audit whether the rendered robot action still matches the original manipulation intent.
Two input paths feed the pipeline. Path A takes datasets with hand-pose annotations, Path B handles raw video by estimating poses with WiLoR plus DynHaMR and using Qwen3.5 to cut long recordings into subtasks with natural-language labels. Four sources (ANT 7h, EgoDex 732h, ViTRA 249h, EgoVerse 954h) total about 1,940 hours; after rendering across 15 morphologies and curation this becomes 18,561 hours. Actions use a camera-frame relative end-effector representation, because camera placement is unknown across sources and cannot be unified into a world frame. The model is a Qwen3.5-4B backbone with a diffusion-transformer action head, pretrained for 200K steps.
The authors decompose RoboTwin2.0 into four independent perturbation axes (visual appearance, scene layout, morphology, task semantics). That decoupling is itself a contribution: earlier benchmarks bundle distribution shifts into one score and hide which factor fails.
Main results (success rate %):
| Pretraining | Clean | Randomized | EBench |
| Robot-only | 62.2 | 50.9 | 39.6 |
| Ego2R + Robot 1:1 | 68.1 (+5.9) | 53.5 (+2.6) | 49.8 (+10.2) |
Ego2R+Robot 1:1 leads five of seven columns, and the 3:1 ratio tops EBench (51.7, +12.1). Broken down, visual appearance benefits most: at 1:1 the model gains +4 on background, +8 on lighting, and +6 on robot color, drawn from scene diversity in the videos and rendering differences across 15 morphologies. On morphology transfer, ARX rises from 44 to 51 and UR5 peaks at 31 (3:1), but Franka stays below 7% (too large a kinematic gap). Task semantics improve steadily: unseen objects climb from 29% to 40% (3:1, +11), and paraphrased-instruction robustness reaches 69% (1:1).
The ablation (ego-only pretraining) shows the pipeline's value most clearly: raw ego video reaches only 28.1%, a single-morphology pass lifts it to 31.7% (+3.6), 15 morphologies to 33.5%, and adding raw ego back as a 16th 'morphology' jumps it to 37.3%. On the real ARX ACone platform across five long-horizon tasks, Ego2R+Robot 1:1 pretraining plus a converted ego-play clip wins all five, with +14 on block-placing and +13 on screw-insertion over robot-only.
Honestly, the absolute gains are modest (+2.6 to +5.9 on RoboTwin), and the 18,561-hour figure is the same 1,940 hours of human behavior rendered across 15 morphologies, not 18,561 hours of fresh behavior. The real value is twofold. First, the synthesized data buys out-of-distribution robustness: gains concentrate on visual, morphology, and semantic perturbations, meaning it teaches invariance rather than just covering more trajectories. Second, this data is near-free and scalable; recording seven minutes of casual hand video and converting it into training data consistently helps, which is practical for anyone short on robot data. The decoupled evaluation protocol is reusable on its own.
The authors acknowledge three points: retargeting maps to a parallel gripper and discards finger articulation, so dexterous hands are out of scope; visual alignment relies on inpainting and depth compositing, which introduce artifacts under heavy occlusion or complex lighting; and evaluation is limited to the RoboTwin2.0 task scope.
A caveat worth adding: the headline 18,561 hours is a 9.6x amplification, so the underlying behavioral diversity is still the 1,940-hour pool; Franka stays under 7%, showing cross-morphology transfer still breaks on large kinematic gaps; and the base-pose search assumes a single arm per scene. Gains on RoboTwin itself are mild, with most of the lift landing on EBench and the perturbation axes, so results need to be read by setting, not in aggregate.