CoorDex: Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation
Sikai Li, Shuning Li, Zhenyu Wei, Yunchao Yao, Chenran Li, Mingyu Ding
cs.RO, cs.AI, cs.LG
2026-06-23
CoorDex splits body and hand into two latent priors plus a coordinated residual, letting a 49-DoF humanoid grasp while walking; WalkGrab hits 0.55 vs 0 for joint-space PPO.
Humanoids cannot grasp well while walking. Existing humanoid loco-manipulation is a stop-and-go process: walk to the object, stop, manipulate, walk again, and it usually relies on low-DoF grippers that amount to an open-close primitive. Move to a 20-DoF dexterous hand and stable grasping during walking gets hard, because wrist pose is set by foot timing, torso posture, and whole-body reaching. If the hand model also has to explain that 6D wrist pose, it spends most of its capacity placing the wrist instead of coordinating fingers. CoorDex targets high-DoF dexterous manipulation during motion.
It converts high-dimensional body and hand control into coordinated latent residual control, built in two parts.
First, two separate priors. A body prior covers the G1's 29 body joints in a 16-dim latent; a hand prior covers the 20 finger joints of the WUJI five-finger hand in a 12-dim latent. Both start from privileged teacher trackers and distill into proprioception-conditioned priors with frozen decoders. The hand prior's key design is wrist-stabilized: during training the reference wrist pose is written directly into simulation, so the teacher controls only finger motion, and the hand latent never has to explain wrist placement. Learned latent commands then serve finger coordination directly.
Second, a coordinated latent residual policy. Each step the two frozen priors encode their proprioception into latent means as the default action. A shared coordination trunk fuses body state, hand state, task context, hand-object state, and the previous residual into a task-level feature. Two separate residual heads then predict body and hand residuals, added to the prior means before decoding into joint targets. The body residual adjusts stepping, torso motion, reaching, and wrist placement; the hand residual refines finger preshape, closure, and contact. Shared context plus separate heads couples the subsystems through task state without collapsing into a single action head.
Exploration happens in a 28-dim latent space (16 body plus 12 hand), not the 49-dim joint space, which is what makes this contact-rich task trainable. Longer tasks use NoDemoRSI, a demonstration-free reference state initialization that bootstraps a reset distribution from states the policy itself visits.
The platform is a Unitree G1 with a 20-DoF WUJI five-finger hand, evaluated over 50,000 episodes in Isaac Lab.
| Task | Success | Note |
| WalkGrab (walk-grasp-carry) | 0.55 | about 0.25 m/s near the bottle, no stop |
| OpenFridge (open fridge while moving) | 0.66 | door to 57.76/60 deg |
| WalkPickTurn (walk-pick-turn) | 0.89 | min heading error 9.98 deg |
Ablations on WalkGrab, same PPO budget:
| Method | Success | Grasp rate | Stops? |
| All joint-space PPO | 0.00 | 0.00 | yes (twists body to avoid falling) |
| Body prior plus joint-space hand | 0.00 | 0.01 | yes (collapses to stationary grasp) |
| Monolithic latent head | 0.00 | not reliable | jittery, action rate 0.40 |
| CoorDex | 0.55 | 0.55 | no |
All three baselines fail; CoorDex is the only one that does not stop and grasps reliably. The monolithic head reaches the interaction region but produces unnatural, jittery motion (action rate 0.40 vs CoorDex's 0.22) and zero reliable grasping.
For humanoid manipulation, this shows grasp-on-the-move is trainable with high-DoF dexterous hands, with no need to fall back to stop-and-go or open-close grippers. Splitting body and hand into two priors and coordinating them, rather than one high-dim action head, is the structural reason a 49-DoF contact-rich task trains at all. The same body prior can also be reused across tasks; swapping the hand or task mostly retrains the residual policy. The gap to real hardware remains perception.
The authors concede three points. The policy uses privileged state observations (object pose, contact signals) and does not yet address perception or visual sim-to-real. Experiments use a single fixed G1 with a WUJI hand, with no evaluation across more objects, hands, or hardware. Longer-horizon tasks still depend on task-specific exploration support like NoDemoRSI. The 0.55 WalkGrab success is in simulation with privileged observations, so real-world reliability would likely be lower. This is a simulation capability proof, not deployment-ready.