Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy
Tara Sadjadpour, Siming He, C. K. Wolfe, Haozhi Qi, Lea Wilken, S. Shankar Sastry, Claire Tomlin, Jitendra Malik
cs.RO
2026-09-24
Morphology- and contact-aware retargeting plus residual RL yields 89.3% zero-shot success over 300 real trials on three- to five-fingered hands.
Human hand-object interactions are plentiful demonstrations for dexterous robots, but transfer stalls on three gaps: morphology (three, four, or five fingers, with different proportions), dynamical feasibility, and sim-to-real. Prior pipelines often split kinematic retargeting, physics refinement, and visuomotor distillation. Several dynamics methods also constrain the human to slow, robot-friendly grasps, or they fine-tune later with teleop. Naturally occurring tabletop motion, especially near-table collision, does not obey those constraints.
Morphometric Imitation, from Berkeley, chains three stages: morphology- and contact-aware kinematic retargeting, residual RL for dynamics, and distillation into a zero-shot visuomotor policy.
Stage one is Morphometric Optimization (MMO). MANO is scaled in palm and per-finger length to match the robot, then optimized for joint and fingertip alignment. Changing morphology drifts contacts, so each frame is re-optimized to recover reference contact vertices (5 mm threshold, the GRAB recommendation) with a table-penetration penalty. Fewer-finger robots get a coupled-finger spacing term. Linear-blend-style weights then skin the aligned MANO skeleton onto the robot skeleton, and arm-plus-hand IK is solved jointly. MMO runs at about 180 fps per hand on an RTX 4090.
Stage two is residual RL in ManiSkill. The policy outputs Δq; commanded joints are the kinematic reference plus that residual. Observations, rewards, and terminations all see object pose and contact. Reward is the product of an ADD object-tracking term and a contact-count term. Table collision or excessive contact force ends the episode immediately. Domain randomization covers anisotropic object scale, mass, friction, a 10 cm × 10 cm table region, and ±15° yaw.
Stage three distills teacher rollouts into a point-cloud student (ManiFlow encoder, DiT-X, flow matching) at 10 Hz, executing three actions before replanning. Hardware is a KUKA iiwa14 with a Sharpa Wave hand and a RealSense L515, on a rigid hardwood table.
Kinematics, ten GRAB trajectories on three hands, location-aware contact F1:
| Hand | MMO F1 / patch | Best baseline F1 / patch |
| Dex3 (3 fingers) | 38.0% / 7.5 mm | 10.2% / 18.9 mm |
| Allegro (4 fingers) | 30.3% / 9.1 mm | 22.0% / 18.7 mm |
| Sharpa (5 fingers) | 37.2% / 9.7 mm | 26.7% / 10.7 mm |
F1 gains over the strongest baseline are 27.8, 8.3, and 10.5 points.
The same residual RL, swapping only the kinematic reference, reaches task success 82.5% / 69.9% / 91.8% versus 53.2% / 68.2% / 56.5% for the best baseline, up to 35 points on Sharpa. Dropping contact or object pose hurts (Dex3: 82.5% with both, 51.2% pose-only, 42.5% contact-only). Allegro's thick fingers and tight table clearance trail both the more human Sharpa and the thinner Dex3.
Hardware: 10 categories × 3 physical instances × 10 randomized poses, 300 trials, 89.3% zero-shot success. Lightbulb is 30/30. Torus is 80.0%; tape rolls are much thinner than the bagel-shaped reference. The transparent wineglass is invisible to LiDAR; with ping-pong balls in the bowl it still hits 93.3%. In simulation the teacher is at 91.8% and the student at 93.8%, so distillation does not dump performance.
If contact geometry is wrong, residual RL spends its budget fighting a bad reference. MMO writes morphology and contact into the kinematic objective instead of hoping dynamics will invent them. One hyperparameter set across three hands is evidence this is not a single-embodiment tune. Zero-shot hardware without teleop fine-tuning is a checkable step on the sim-then-real path. It is incremental: the data are still ten mocap trajectories.
Residual RL and visuomotor policies are trained per object category, and hardware uses only KUKA+Sharpa. Demonstrations are GRAB mocap meshes, not monocular reconstructions; the scaling story assumes video later, and this paper does not ingest it. Anisotropic scale randomization does not cover thickness jumps like the torus. Ping-pong balls in the wineglass are an evaluation patch. Table collisions on hardwood are expensive, so the success rule that forbids hard hits is fair, and it also shows the workspace is still controlled.