Open-HumanWAM: An Open Recipe for Scaling Human and Robot Experience for Embodied Foundation Models
Juncheng Ma, Jiaxin Li, Jiankai Tu, Bo Liang, Guodong Lv, Yufan Deng, Junzhe Tang, Liangzhu Yang, Chen Bian, Shukai Gong, Yintianrun Zhang, Zicong Cao, Baohao Zhu, Jianxin Bi, Tengfei Yan, Yinyuan Zhang, Kewei Zhang, Xinglong Chen, Kaiqi Chen, Duomin Wang, Xiaotian Tang, Liang Tan, Li Li, Yuan Xu, Haosen Yang, Yuqi Wang, Jingfei Xia, Songyan Huang, Zhiyang Dou, Zhen Dong, Enze Xie, Tat-Seng Chua, Daquan Zhou
cs.CV
2026-11-05
A PKU-led team open-sources a 50k-hour egocentric video corpus and full training recipe; the pretrained world-action model sets SOTA with 88.65% average on RoboTwin 2.0 and lifts real-robot OOD success from 2.5% to 67.5%.
Robot foundation models are bottlenecked on data. Teleoperated trajectories are expensive and narrow in scene coverage, while egocentric human video, footage from head-mounted cameras, is cheap, abundant, and spans kitchens, workshops, and labs. Turning that video into robot priors requires answering three questions: which clips are worth training on, how human hand motion maps to robot actions, and how much closed-loop success the pretraining actually buys. Prior work (EgoScale, HumanScale, Dyna-2) established the value of egocentric pretraining, but each used its own recipe, some judged transfer only by held-out action loss, and none shipped a reproducible open pipeline scaled to tens of thousands of hours. Open-HumanWAM, led by PKU with NUS, MIT, UCSB, and NVIDIA, fills the gap with a fully open stack: a 50k-hour corpus, curation and training pipelines, checkpoints, and evaluation suites.
The corpus is curated from 104k hours of raw video down to exactly 50,000 hours (35.9M clips). A HumanNet internet-video subset contributes 40,744 hours; EgoSuite-Open100K, Gen-HumanEgo, EgoDex, and EgoVerse supply the rest. Curation targets three failure modes: hands out of view or clipped at frame boundaries, jumpy hand and camera pose estimates (segments cut at smoothness, acceleration, and jerk thresholds), and long idle stretches (detected with a 32-frame sliding window and trimmed).
All trajectories are aligned into a 16-D bimanual end-effector space: 3-D position, quaternion, and gripper openness per hand, expressed in the camera frame, with wrist axes standardized via Procrustes alignment. This step matters: on the same 1,000-hour four-source mix, aligned actions reach 52.2% scene-OOD success (background and texture swaps) versus 43.2% unaligned.
HumanWAM-Vanilla attaches an action expert of equal depth to the Wan2.2-TI2V-5B video backbone, forming a Mixture-of-Transformers that jointly denoises future video and 32-step action chunks. It uses inverse-dynamics (IDM) attention: during training, action denoising is conditioned on noised ground-truth future video; at inference the model generates video first, then predicts actions. Ablations favor IDM attention combined with absolute camera-frame poses.
HumanWAM-Goal co-trains the 50k human hours with about 14.6k hours of robot data and adds goal-image conditioning: given a picture of the desired end state, the model predicts actions that reach it. The action space grows to 49 dimensions, and gripper openness becomes a physical aperture normalized to a 0.12 m reference, putting human hand opening and robot gripper width on one metric scale.
Downstream success grows log-linearly with pretraining hours: SR = a + b·log10(H). Under isolated attention, scaling from 100 to 10,000 hours moves clean success from 71.2% to 80.0%, scene-OOD from 25.2% to 58.1%, and random (full randomization of background, lighting, clutter, table height) from 8.4% to 28.7%, against 69.5/16.2/2.1 without egocentric pretraining. Fits give R² of 0.971, 0.949, and 0.877; every 10× more hours buys 15.88 pp on scene-OOD but only 4.43 pp on clean, so the gains concentrate out of distribution. Under IDM attention the trend holds from 100 to 50k hours: scene-OOD 49.1%→73.2%, random 21.5%→42.9%.
Diversity beats duration. At a fixed 5,000 hours, corpora whose task and scene coverage grows with scale show a fitted scene-OOD slope 4.73 higher than corpora that only add trajectories within fixed coverage. Human data also beats robot data: from 500 to 5,000 hours, egocentric pretraining wins in every setting, and 1,000 hours of human data outperforms 5,000 hours of robot data.
On RoboTwin 2.0, after post-training on 50 clean demonstrations per task:
| Method | Clean2Clean | Clean2Random | Average |
| π0.5 (VLA) | 70.70 | 46.00 | 58.35 |
| GO-3 Nano Preview | 88.54 | 81.48 | 85.01 |
| PICA-WAM | 90.68 | 74.70 | 82.69 |
| HumanWAM-Vanilla (human-only) | 92.08 | 68.88 | 80.48 |
| HumanWAM-Goal (hybrid) | 92.70 | 84.60 | 88.65 |
On a physical Cobot Magic bimanual robot, four tasks (cup stacking, towel folding, drawer storage, desk tidying) under tablecloth and lighting changes: the baseline without egocentric pretraining succeeds once in 40 out-of-distribution trials (2.5%); HumanWAM-Vanilla averages 67.5%. HumanWAM-Goal runs 20 real-world tasks zero-shot; holding robot data fixed at 14.6k hours and adding 45.9k human hours lifts mean success from 20.0% to 30.5% and task coverage from 10/20 to 16/20. The same frozen policy changes behavior when shown a different goal image.
Three practical takeaways. Egocentric human video is a cheaper pretraining fuel than teleoperation: 1k human hours beat 5k robot hours. The scaling law gives budget guidance: every 10× data buys double-digit OOD gains, and expanding task and scene coverage yields steeper returns than piling on hours. And this is the first fully open egocentric scaling pipeline, with corpus, code, and checkpoints released, so other groups can keep scaling from 50k hours. To be clear about the contribution: the world-action architecture itself is deliberately basic, as the authors state; the value here is the recipe and the evidence, not model design.
The authors concede two: on RoboDojo the 14.17% mean success trails the strongest models, partly because some tasks need memory and arithmetic that a basic architecture lacks; and Vanilla's Clean2Random (68.88) trails GO-3 Nano (81.48), attributed to missing specialized modules. Further concerns from a close read: the scaling law is fit only on the RoboTwin simulation suite, and whether it stays log-linear beyond 50k hours is untested; real-robot comparisons include only the No-Ego baseline, with no head-to-head against π0 or GR00T under identical settings; about 81% of the 50k hours come from HumanNet internet video with estimated wrist poses, and annotation noise is never quantified; most RoboTwin baselines are taken from the official leaderboard rather than re-run in the authors' environment.