Hydra-0: Action Flow for Generalist World Modeling and Control
Hongyu Li, Bowen Wen, Xinghao Zhu, Yixuan Wang, Yilun Du, Yunzhu Li, George Konidaris, Stan Birchfield, Soha Pouya, Chenran Li, Yan Chang
cs.RO
2026-08-19
Hydra-0 conditions world models on image-plane action flow, cutting gripper EPE 90.4% and object EPE 60.2% vs a 6D Cosmos baseline; RoboLab open-loop r=0.96.
A robot world model should answer: if this command runs, how will the arm and the object move in the camera. Most action-conditioned video models stay tied to the training embodiment. Joint commands encode kinematic structure; the same end-effector pose traces different link paths on different robots. The video model then learns an embodiment-specific map from commands to pixels, and transfer across arms, bimanual setups, and human hands breaks.
Hydra-0 writes the action as sparse image-plane trajectories, action flow: pixel paths plus visibility for points on the visible robot surface or on the manipulated object. The format does not expose joint space, so arms, grippers, and hands share one condition. The paper is a NVIDIA / Brown / Columbia / Harvard collaboration. The target domain is robot interaction with deformables, where explicit state and physics remain brittle.
Two construction routes. When geometry and camera calibration exist, Isaac Lab rolls a candidate command through the controller and physics, projects link-surface points into the camera, and marks a point visible only if depth is positive, in-frame, and consistent with the depth buffer. For public videos without URDF or extrinsics, AllTracker yields dense tracks, then grounded masks assign them to embodiment versus object. Training samples four modes: embodiment only, object only, all tracks, or none (dropout). At deployment, forward prediction builds the condition from a command rollout, never from future video.
Conditioning follows ATI and Wan-Move: first-frame latent features are Gaussian-propagated along trajectories onto future latent cells, with a presence gate telling the backbone where motion is specified. The same interface is wired into Cosmos 2.5 and Wan2.2 (I2V-A14B and TI2V-5B); only the injection layer changes. Backbone weights stay frozen except the DiT patch embedding and rank-64 LoRA. The corpus covers seven sources, about 178,187 episodes, and 2,201.7 hours after filtering, mostly cloth, cable, bag, and paper. Wan2.2 A14B trained 40,000 steps over five days on 32 H100s.
Long-horizon rollout uses LongLive-2.0-style causal chunks with a KV cache, then DMD2 distillation to four steps per chunk. Forward mode feeds gripper flow for open-loop policy replay. Inverse mode supplies only desired object flow; a world action model infers compatible robot motion, and a lightweight head reads DiT tokens into executable commands. Paired real trajectories suffice; expert demonstrations are not required.
Five held-out sets, 100 clips each, shared horizon. The best reported configuration is the four-step distilled Wan2.2 A14B. Against a Cosmos 2.5 baseline fine-tuned on native relative 6D end-effector actions, mean gripper EPE falls from 34.28 to 3.29 pixels (about 90.4%) and object EPE from 13.23 to 5.27 (about 60.2%). PSNR, SSIM, FID, and FVD move in the same direction. Swapping only the condition on the Cosmos backbone already cuts gripper EPE from 34.28 to 13.80, so the interface, not only a bigger video backbone, carries much of the gain. ATI and Wan-Move transfer zero-shot with plausible motion, but lag on object EPE and FVD.
| Method | Gripper EPE↓ | Object EPE↓ | FVD↓ |
| Cosmos 2.5 (relative 6D) | 34.28 | 13.23 | 405.8 |
| Hydra-0 (same backbone + action flow) | 13.80 | 6.27 | 277.4 |
| Hydra-0 (Wan2.2 A14B 4-step) | 3.29 | 5.27 | 155.9 |
After multi-embodiment mid-training, six IWS tasks at 0% target data already beat a run started from raw Wan2.2 weights. By 20% task data, later changes stay within 3.4% LPIPS, 6.7% flow EPE, and 6.8% FVD. On one H100, 81 frames at 480×832, guidance and VAE off, the four-step student hits 61.98 fps, 16.0× the bidirectional teacher on the generation stage (decode excluded).
Open-loop RoboLab evaluation covers five policies, six tasks, ten rollouts per pair, 300 episodes. Replayed versus reference success rates have Pearson r=0.96, Spearman ρ=0.93, and MAE 5.7 percentage points; task-averaged rates also keep the five-policy ranking. Real cloth-folding replays of successes and failures qualitatively keep the outcome. Inverse control is shown on one flexible-pipe-bending task: object flow from a held-out human demo, predicted robot motion, executed commands. No task-level success table.
If robot motion is written as camera-visible tracks, heterogeneous interaction data can share a trainer, and open-loop evaluation and inverse control can share an interface. That is especially useful for deformables, where reconstructed state is unreliable and pixel motion is the supervision everyone already has. Teams already generating action-conditioned video with Cosmos or Wan can try swapping the condition before swapping the backbone; this paper puts numbers on that swap.
The r=0.96 figure is a replay test: given the executed trajectory, does the world model still produce the same success or failure. It is not closed-loop control on the model's own frames. Inverse control is a proof of concept, not a general imitation-learning recipe.
The world action model can miss grasps by centimeters; the authors point to weak depth. Generated rollouts sometimes leave contact state ambiguous. Wrist-camera work is a qualitative DROID demo. Evaluation stops at open loop. Inverse control is one pipe-bending task without baselines or success rates. The corpus is deformable-heavy; rigid objects, mobile manipulation, and large camera motion are not systematically measured. The 62 fps number excludes VAE decode, so end-to-end is slower; the robot action head still uses the 50-step teacher.