XPACE: Joint World and Action Modeling from Heterogeneous Experience
Jiacheng Wei, Jerry Bai, Xiaoyu Yue, Zidong Wang, Xiaoyang Guo, Cheng Chen, Fanqi Pu, Fan Wu, Zhixu Yue, Yizhuo Li, Feng Qiu, Bo Liu, Yuying Ge, Hui Zhou, Chenyi Chen, Yixiao Ge
cs.RO
2026-09-16
XPACE jointly predicts actions and future video from mixed human-robot data, then fine-tunes on simulated recoveries. On XPENG's IRON, pouring success rises from 50% to 95%.
A general-purpose humanoid is missing two kinds of supervision at once. Human egocentric video covers many scenes and skills, but its labels stop at pixels or body kinematics, not robot commands. Robot teleoperation supplies executable actions, and collecting it across rooms and objects is expensive. Mixing both into one dataset does not yield a model that can emit actions, roll out the visual future of a prescribed motion, and then train on those rollouts.
Imitation has a second hole. Expert demonstrations are almost all successes. Once closed-loop execution leaves that manifold, the policy has barely seen how to come back. Classical DAgger asks a human to label those states. XPACE, from XPENG Robotics, tries to close both gaps with one video backbone: learn control from mixed human and robot experience, then use the model's own simulator to synthesize recovery trajectories.
The corpus is a four-layer pyramid of about 5,000 hours of embodied video. L1 is in-the-wild egocentric video with no action labels, used only for visual dynamics. L2 is human video with hand and wrist poses. L3 is task-aligned and appearance-aligned bridge data, overlapping tasks and camera views without an explicit feature-alignment loss. L4 is teleoperation on IRON. Failures, low-progress clips, and recoveries form a separate set F that trains the simulator only. Human and humanoid motion share a kinematic schema of end-effector poses, hand articulation, and torso plus camera, with embodiment-specific linear projections at the action-head boundary.
The network is an asymmetric mixture of transformers: a heavier video stream, and an action stream that only reads video features. The video backbone starts from Wan2.2-TI2V-5B and predicts in chunks of four latent frames, sixteen video frames. In policy mode, visual history, current state, and a language instruction produce future video and a 16-step action chunk. In simulator mode, language and the action head drop out; a prescribed 2D skeleton sequence and camera poses condition future video. Skeleton tokens are added to aligned video tokens, with a zero-initialized embedding so pretrained features survive the first step. Fifteen action blocks read dense features from video blocks 16 through 30. Actions cannot rewrite video in the forward pass, but action-loss gradients flow back into the backbone.
Stage I trains video prediction only. Stage II samples policy and simulation updates 50/50, shifting the mixture from broad human and unlabeled video toward the target robot, while Phase II-c still replays human and bridge data. Captions move from dense descriptions toward the short task instructions used at deployment. F enters in II-b, simulation only.
Post-training turns the simulator into a recovery factory. Stage III-sim applies chunk-wise self-gradient forcing: roll out context without gradients, detach and re-encode the generated prefix, then fit ℓ2 flow matching against recorded targets. Sampling drops from 50 ODE steps to 8. Distribution matching is avoided because the simulator must follow the supplied controls, not merely look plausible. Stage III-policy fine-tunes a separate copy of the II-c policy. Around an expert trajectory, the arm with the largest end-effector travel is displaced along a direction 90° to 150° from the next expert motion, then returned. The simulator renders only the inserted 2k frames; the prefix and expert tail stay real. DINOv3, depth-weighted PSNR, and SSIM must show both relative recovery and absolute consistency with the pivot. Filtered recoveries are 8% of the fine-tuning mix. The stage finishes in under a day.
Real-robot tests run on IRON-R01-1.11, 20 trials per task per method, against GR00T and DreamZero. Teleoperation and action-labeled human data are matched; only XPACE gets Stage I video adaptation. The three-task suite treats banana pick-and-place and pouring as in-distribution. Bowl stacking is absent from L4 and present in human data, so it tests human-to-robot skill transfer.
| Method | Mean progress | Mean success |
| XPACE | 0.84 | 68.3% |
| DreamZero | 0.68 | 40.0% |
| GR00T | 0.36 | 6.7% |
Banana succeeds in 16/20 trials, including all five preset positions. Pouring is 10/20 with progress 0.78. Stacking is 15/20. Pouring is the hardest task for every method and the widest gap versus DreamZero (1/20). Progress 0.78 with 50% success means many trials rotate the bottle without finishing the pour. On short-horizon video prediction, token addition beats AdaLN by 0.95 dB PSNR in-distribution and 1.11 dB out-of-distribution.
Full data versus robot-only: banana success across five positions rises from 50% to 80%, progress from 0.70 to 0.93. With unseen distractors (cucumber, flower, toy duck), success rises from 30% to 50%, progress from 0.48 to 0.74. After matched action fine-tuning, Stage I video pretraining cuts evaluation action loss by up to 12.5%; the first eighth of exposure already yields 8.1%. Keeping human supervision during robot adaptation cuts loss 14.0% versus robot-only; human mid-training then robot-only fine-tuning yields 8.5%. On robot-absent, human-dense behaviors, co-training cuts loss 22.7%.
SGF lifts 193-frame in-distribution PSNR from 16.94 to 17.83 and speeds inference about 3.00×. DAgger fine-tuning is scored on banana, pouring, and cola handover. Mean progress goes from 0.81 to 0.93, mean success from 61.7% to 86.7%. Pouring jumps from 50% to 95%, progress 0.78 to 0.98. Banana goes 80% to 90%; cola handover 55% to 75%.
On a humanoid whose arms roughly match human kinematics, human motion labels can supervise the same action objective as robot commands. A shared video backbone lets unlabeled video, human demos, and teleoperation train one dynamics representation, which is what lets the simulator render off-trajectory states from a skeleton. An 8% mix of synthetic recoveries moves pouring from half-success to near-complete. That points at closed-loop drift as the bottleneck, not a shortage of teleop hours.
The portable piece is the recipe: the data pyramid, coarse-to-fine replay, SGF, and filtered recoveries. This is not a drop-in open-weight model. The external comparison measures full recipes, not architecture in isolation.
The paper is explicit: the DAgger comparison is before versus after fine-tuning, and does not separate recovery data from an extra day of optimization. The external baselines are not an architecture-fair match. Bridge data is not ablated on its own. The conclusion still sits in demonstration-centered recovery; uncertainty, longer contact-rich interaction, and real-world feedback without forgetting remain future work.
The eval is narrow. Twenty trials per task, five preset banana positions. Policy-simulator rollouts already catch target errors: a red fruit for a green-fruit instruction, a bottle for a curved-fruit instruction. Failure set F never trains the policy, so off-nominal states for control come almost entirely from synthetic recoveries. No code or weights are released.