Learning Agile Perceptive Traversal of Sparse 3D Structures for Humanoids
Efe Ongan, Chong Zhang, Boyang Sun, Andrei Cramariuc, Cesar Cadena, Marco Hutter
cs.RO
2026-08-30
An ETH PM-01 with passive hooks and a head-mounted solid-state lidar completes jump-up, brachiation, and jump-down in 14 of 15 hardware trials, reaching 0.5 m/s.
Legged controllers usually sit on an intermediate map. Elevation maps are cheap and drop thin, overhanging geometry. Voxel grids keep 3D structure and get expensive as resolution rises. Policies that eat raw depth or point clouds already handle dense ground and bulky obstacles. Accurate contact on centimeter-scale sparse structure is still rare.
Monkey-bar brachiation is the stress test: thin bars, sparse contacts, and explosive whole-body motion at once. Earlier brachiators were two- or three-link special-purpose machines, or they assumed known bar poses. LadderMan climbed ladders with a depth policy. None of those systems ran jump-up, brachiation, and jump-down from onboard sensing on a general-purpose humanoid. ETH Zurich's Robotic Systems Lab does that on an EngineAI PM-01.
The hands come off. A water-jet-cut stainless-steel hook takes their place. The opening fits a 60 mm circle; training bars have 1 to 3 cm radius, so placement error in the swing plane has room. Release is a wrist-yaw rotation that turns the hook out of the bar plane. The body does not have to lift, so arm torque and heat stay lower. Symmetric geometry also allows backward travel.
The head carries a RoboSense E1R solid-state lidar: 192×144 native scan, 120°×90° field of view, built-in IMU. Electronic scanning distorts less than a spinning lidar when the torso accelerates hard. The policy sees a decimated grid, 36×35×4 for brachiation and 36×48×4 for ducking, at 10 Hz. Each cell stores hit position relative to the sensor plus range. Brachiation crops to the upper workspace; ducking keeps the full field.
Training has two layers. Three privileged MLP teachers for jump-up, brachiation, and jump-down learn in IsaacLab with PPO. Teachers see ground-truth bar endpoints plus contact, base velocity, battery, and thermal state. The brachiation teacher tracks a planar goal; the jump teachers use contact rewards and no human motion clips. Terrain is a procedural ladder with randomized geometry plus clutter. Jump skills use a height curriculum; brachiation widens spacing and thins the bars.
The student sees only onboard signals. Distillation switches the active teacher by phase, in three stages. Stage one is DAgger behavior cloning. Stage two warms the critic on unclipped GAE while cloning continues, so the jump into RL does not blow up. Stage three is regularized PPO: the clipped surrogate plus a decaying cloning anchor, so the student can deviate where its perception is worse than the teacher's. Reward scales differ by phase, so advantages are standardized per phase and the value loss is weighted by inverse return variance.
The perception backbone is AME-2's attention encoder, retargeted from elevation maps onto the lidar grid. The cloud stays on its native 2D raster rather than being treated as an unordered set. Features go into a GRU with proprioception (4-frame history) and the command. An MLP decodes 23 joint-position targets at 50 Hz. During distillation an auxiliary head predicts the nearest bar's relative pose from the GRU state.
The same AME-2 plus GRU trains a second ducking policy. That teacher sees a height map with an upward raycast channel. The student still sees only raw E1R returns.
Sim-to-real hangs on three models. Jump-up peaks at 2.11 kW mechanical joint power and sags the pack to 34.7 V. The 18 low-torque actuators, including 10 arm joints, hit 93% of limit during those sags. Without the voltage model, jump-up browned out the robot. With a voltage-sag penalty, no hardware trial browned out. A leaky thermal integrator on those actuators saturates at a sustained load ratio of 0.2. Lidar noise is dominated by a 0.625° beam cone that bleeds at depth edges. Training uses a Warp raycaster in IsaacLab with 2 cm range noise. Validation in MuJoCo casts 16 rays per pixel inside the cone.
The encoder ablation reports behavior-cloning loss after the first distillation stage (table values ×10⁻², lower is better). The attention grid encoder has 13.8k parameters, about 8× fewer than the CNN (106.7k) and two orders fewer than the MLP (1.31M), and it wins every phase. Adding a bar-centerline auxiliary loss cuts the loss further. A blind student is worst: proprioception alone cannot do this task.
| Encoder | Params | Total | Jump-up | Brachiation | Jump-down | Centerline |
| AME-2 + aux | 13.8k | 2.35 | 2.48 | 1.95 | 3.20 | 0.71 |
| AME-2 | 13.8k | 2.43 | 2.56 | 2.00 | 3.26 | 1.62 |
| CNN | 106.7k | 2.62 | 2.73 | 2.16 | 3.57 | 1.79 |
| MLP | 1.31M | 2.76 | 2.92 | 2.28 | 3.62 | 1.85 |
| Blind | – | 2.90 | 3.18 | 2.30 | 3.51 | 2.07 |
On hardware the robot runs the full jump-up, brachiation, jump-down sequence on three ladders. An operator supplies one planar goal in the odometry frame; pose comes from SE(3) lidar-inertial odometry. 14 of 15 trials succeed (93% full-sequence). Peak brachiation speed is 0.5 m/s. The one failure was a missed next-bar catch after a successful jump-up.
| Setup | Height (m) | Spacing (m) | Jump-up | Brach. | Down | Full | n |
| Ladder A | 1.69 | 0.26 | 100% | 100% | 100% | 100% | 9 |
| Ladder B | 1.72 | 0.31 | 100% | 100% | 100% | 100% | 2 |
| Ladder C | 1.75 | 0.33 | 100% | 75% | 100% | 75% | 4 |
Sim-to-sim in MuJoCo holds spacing at 0.35 m and sweeps height from 1.65 m to 1.90 m, 10 trials each. Full-sequence success sits between 70% and 90%. Brachiation and jump-down are 100% at every height; the drop is jump-up as the first bar nears the jump limit (90% down to 70%).
Ducking clears cylindrical bars of 1 to 5 cm diameter and 1.1 to 1.5 m underside clearance at 100% in simulation whenever the robot physically fits. On hardware, 2 cm × 2 cm wooden slats in random orientations unseen in training are cleared in 10 of 10 passes, including a 1.2 m clearance example.
Three takeaways for people who ship humanoid controllers. Centimeter-scale sparse geometry does not need a voxel map first: a head-mounted solid-state lidar, attention over the scan grid, and a GRU are enough to support an explosive jump-up. The sim-to-real work that actually moved the needle is the battery and thermal models in the reward. Skip the voltage model and the robot browns out. The hooks are an engineering choice, not a grasping solution. The same perception backbone does train a ducking policy for 2 cm overhead sections, so the encoder is not brachiation-only.
This is a systems paper. Hands are gone, the goal is still human-provided, and the geometry family is narrow. The piece worth copying is raw lidar plus contact-accurate control, not the hook as a product feature.
The authors say the current system is a small set of separately trained task policies, and robustness to much more diverse geometry is unproven. Direct-lidar perception on partially observed structures that need long-horizon spatial memory is listed as future work.
The 15 hardware trials are uneven: Ladder B has 2 runs, Ladder C has 4 and loses one brachiation, so the 93% figure sits on a small denominator. The failure mode is a hook that does not advance, and contact tolerance comes from a 60 mm opening, not from grasping. Training bars span 1 to 3 cm radius and 0.25 to 0.5 m spacing; messier scaffolding is untested. The training lidar range gate is 0.3 to 1.5 m, so farther bars are outside this paper.
The "first demonstration" claim covers the combination of a general-purpose humanoid, onboard sensing, and the full jump-up–brachiation–jump-down sequence. Purpose-built brachiators and systems with known bar poses sit outside that sentence.