Fingers as Legs: Learning Self-Supported Locomotion and Manipulation with an Anthropomorphic Hand
Amirhossein Kazemipour, Hehui Zheng, Robert Katzschmann
cs.RO, eess.SY
2026-09-15
An unmodified 818g WUJI hand crawls on 14 surfaces, recovers 21 of 25 falls, types 29 of 32 keys without vision, and pushes a cube at 17 mm mean error.
A hand bolted to an arm can only work where the arm can follow. Cabinets, slits, and undersides of tables are typical places where the arm stops and the hand becomes dead weight. ETH Zurich's Soft Robotics Lab wants the hand to walk in on its own, do the job, and walk back for pickup. The same five fingers have to locomote, hold up the body, and make contact with the world. No extra mobility hardware.
A stock anthropomorphic right hand is a bad quadruped. The WUJI hand has an opposed thumb and four fingers of different lengths, so left-right mirroring does not exist. The palm sits at a tilt, so the body frame is not a level command frame. Lift one fingertip to step or press a key, and the remaining contacts must carry the whole 818 g robot. Earlier crawling "hands" usually changed the hardware: DLR-Crawler used six identical finger modules as a hexapod; Gao et al. built a detachable, symmetric, reversible hand. This paper keeps the commercial WUJI kinematics and the factory position controller.
The 738 g stock hand gets an 80 g dorsal pack: Raspberry Pi Zero 2 W, BNO085 IMU, four-cell LiPo. Total mass is 818 g. Twenty joints, four per finger, non-backdrivable. The firmware low-pass on position commands is set to 3 Hz; each joint is current-limited to 1.0 A. The policy runs as 32-bit ONNX on the Pi at 50 Hz, with a 500 Hz serial driver. Crawl commands come in on a dedicated 2.4 GHz gamepad. Wi-Fi stays out of the control loop.
Training is PPO in NVIDIA Isaac Lab. Every policy sees 46 proprioceptive numbers: 20 joint angles relative to a fixed reference, gravity direction and angular velocity in the palm frame, and the previous action, each with an 8-step history. The network outputs a joint-target increment that the stock position loop tracks. Angle scales are 0.060, 0.028, and 0.040 rad for crawl/recovery, keyboard, and pushing. The simulator is fitted to hardware: joint stiffness about 18.7±2.5 times the uncalibrated model, kinetic fingertip friction 0.88, closed-loop delay about 19 ms, filtered flexion/abduction speeds 2.8/2.5 rad/s.
The crawl reward is written around this hand's stance, not around a timed gait chart.
Yaw tracking is added only after a forward gait appears, so the policy does not learn stepping and turning at once. Fall recovery is a second policy: episodes start on the side at a random heading and reward reaching crawl-stance height and joint pose. In simulation, 28 of 32 falls righted within 20 s, median 6.1 s.
Keyboard and pushing get their own policies. Keyboard has no vision: four learned press locations, aligned by an operator, plus encoder forward kinematics. Pushing uses an overhead camera on a dorsal marker and the object, trained with 60 Hz sample-and-hold, 1-3 step latency, 3% dropout, and 2 mm position noise.
In simulation the formulation is 0.65 cm/s faster than a tuned quadruped stock reward (12 seeds, 95% CI [+0.26, +1.02]), and faster in 10 of 12 seeds. Dropping the footprint term costs 0.79 cm/s. Tuned stock has a higher mean five-finger participation, but it plants nails more in every seed; non-nail participation is 0.35 higher here. The lift and direction terms show no independent speed gain in the ablations. Removing direction alone raises five-finger participation by 0.31. The hardware policies still include it because that is how they were trained.
| Setting | Metric | Result |
| Ours vs tuned stock | Sim task speed | +0.65 cm/s (faster in 10/12 seeds) |
| Footprint removed | Sim task speed | −0.79 cm/s |
| Untethered crawl | Mean path speed, 21 trajectories | 0.093 m/s |
| Fall recovery | Hardware / sim | 21/25 (84%); 28/32, median 6.1 s |
| Keyboard | Consecutive commands | 29/32 hits in 72.5 s, median latency 0.25 s |
| 40 mm, 41.4 g cube | 15 deliveries, 10–40 cm | Mean final error 17 mm (5–37 mm) |
The hand crawled untethered on 14 surfaces from rubber mat to gravel. That run is a qualitative demo. With no steering command it drifts right at about 6°/s regardless of initial heading. Right-turn rate scales with the command; left-turn rate saturates. An IMU proportional heading loop hits ±15° and a 30° right turn. A 30° left turn falls short, because the offset that cancels the right drift pushes left commands outside the training range.
On the keyboard, 32 mixed arrow commands with no realignment produced 29 hits, maximum tilt 7.7°. All three misses were Up commands that hit Right Shift. Operator-issued sequences completed the optimal 9- and 12-move Sokoban solutions for one- and two-box levels. Pushing counts as delivered after entering a 2 cm radius and staying within 5 cm for 1 s.
This is an engineering sample of training a shelf anthropomorphic hand as if it were a quadruped, not a new algorithm. The position loop and finger geometry stay stock. Sim-to-real rides on measured stiffness, friction, and delay, plus a footprint term written for an asymmetric stance. Mirror losses and timed gait tables from quadrupeds do not apply to an opposed thumb.
The copy-paste surface is small. Keyboard needs a person to align it. Pushing needs an overhead camera. Crawling needs a human on the gamepad. The intended use is a larger robot dropping the hand at a restricted opening, letting it walk in, press, push, and walk back. Onboard perception and a wider heading range are still missing.
If the question is how to reward unequal legs, the footprint term is the part worth stealing. The ablations put both the speed and the five-finger-participation gains on it.
The authors flag two follow-ups: a wider commanded heading range, and perception on the robot. Keyboard alignment is still manual. Pushing dies outside the overhead camera's workspace.
Policies are task-specific. There is no single loco-manipulation policy. The 14-surface crawl is video-level; there is no per-surface speed or fall table. Open-loop drift of 6°/s means the left-right asymmetry was not absorbed by the policy. The direction-term ablation does not justify keeping it, yet the hardware policies carry it. Six of the twelve tuned-stock seeds also helped pick those weights, so the baseline is a bit optimistic. Higher footprint weights were swept only in simulation. Hardware used the set locked before the ablation.
The four recovery failures were fingers catching on each other. The three keyboard misses all hitting Right Shift point to a systematic offset between learned press locations and the real keycaps, with no vision to correct it.