Perceptive Behavior Foundation Model: Adapting Human Motion Priors to Robot-Centric Terrain
Zifan Wang, Yizhao Li, Teli Ma, Qiang Zhang, Yudong Fan, Hao Xu, Shuo Yang, Junwei Liang
cs.RO
2026-06-06
Perceptive BFM keeps the raw human clip as command and adapts footholds from a height map. TCRS supervision lifts logged tracking success from 27.3% to 55.1% in simulation.
Humanoid behavior foundation models treat a human motion clip as a unified command: one policy tracks walking, running, dancing, and flips. Trackers in the OmniH2O, HOVER, and SONIC line assume that clip is already physically compatible with the ground under the robot. That assumption fails as soon as the demonstrator, the teleoperator, and the robot do not share a floor. A flat-ground walk does not name stair footholds. A clean-studio demonstration does not name swing clearance over a recessed obstacle. The clip carries intent and style. It does not carry contacts, body height, or timing for the robot's local terrain.
Perceptive locomotion and parkour policies can cross obstacles from height maps. They choose a traversal skill. They do not keep an arbitrary user motion as the thing to preserve. The missing piece is perceptual grounding of a motion prior: the command stays the raw clip, and the robot's own terrain scan decides how that clip can actually be stepped.
Perceptive BFM is from Mondo Robotics with HKUST (Guangzhou), HKUST, USTC's AGI Institute, and Nanjing University. At deployment the policy sees three inputs: the raw kinematic reference, proprioception, and a torso-centered ray-cast height map covering 1.6 m x 1.0 m at 0.1 m resolution (17x11 cells plus a validity mask). Terrain-conformal references are training-only. They are never queried online.
Training has four stages under the name Perceptive Motion Tracking (PMT).
TCRS (terrain-conformal reference synthesis) repairs human clips offline against sampled height fields. Stance and swing labels come from foot height, velocity, and hysteresis. Stance feet latch to support surfaces. Swing trajectories are optimized in a mid-foot frame, the midpoint of toe and heel, with sampling-based MPPI knot updates that penalize penetration, vertical edges, and jerk, while pinning liftoff and landing. Root height is reconstructed from support contacts, then a damped multi-point Jacobian IK over twelve leg joints fits ankle, toe, and heel. Horizontal footfalls stay close to the raw trace. This is local repair, not a global footstep planner.
A blind Transformer teacher tracks the TCRS reference with PPO and never sees terrain. The student receives the original raw reference plus the height map. Teacher and student residuals live in different command frames, so distillation copies the teacher's effective PD target minus the raw joint command (target-frame alignment). During DAgger-style rollouts the teacher-control probability anneals from 1 to 0.
Terrain features enter through two zero-initialized residual pathways, one on the intent latent and one on the action mean, each gated by tanh. At init the student is a raw-reference tracker. PPO fine-tuning scales the inherited backbone learning rate by 0.3 and lets the terrain encoder and residual heads move faster. Control runs at 50 Hz. Teacher PPO uses 5e-4; distillation and vision fine-tuning use 1e-4.
On TCRS itself, before any policy rollout, 30 clips on stepping stones with stairs: penetration falls from 5.48 cm (Z-offset ankle lift) to 2.38 cm, and clearance violation falls from 14.3% (cubic interpolation plus single-point IK) to 7.4%. TCRS does not win every column. Float rate is 32.3% versus 12.4% for Z-offset. Foot smoothness is 8.6 m/s^2 and upper-body deviation 4.00 cm, both slightly worse than cubic+IK (6.9 and 3.98). Near step edges the synthesizer prefers missing a contact to colliding.
Under a shared 10k-iteration PPO budget, end-of-training logged closed-loop success:
| Variant | Success | Anchor err. | Joint err. |
| Full PMT | 55.1% | 0.159 m | 1.243 rad |
| No identity gate | 30.4% | 0.231 m | 1.428 rad |
| Vision concat | 29.9% | 0.234 m | 1.421 rad |
| No TCRS | 27.3% | 0.229 m | 1.833 rad |
| No target-frame align | 26.7% | 0.221 m | 1.959 rad |
| No vision | 26.5% | 0.238 m | 2.464 rad |
Dropping vision, TCRS, or frame alignment lands around 26-27%. Stage ablations that remove distillation train from scratch, so rewards are not comparable. The paper flags the lack of seed variance and of a held-out command-terrain split. Treat this table as a training-window diagnostic.
Reference-source ablation: raw 27.3%, Z-offset 33.0%, cubic+IK 41.0%, TCRS 55.1%. Better geometric repair tracks with better downstream numbers, still without statistical significance.
A separate fixed-policy sim eval runs 450 episodes (5 terrain families x 30 motions x 3 seeds):
| Terrain | PMT complete | No TCRS | PMT shank collision | No TCRS collision |
| Stairs | 53.3% | 24.4% | 11.1% | 26.7% |
| Slopes | 66.7% | 35.6% | 5.6% | 17.8% |
| Sparse supports | 42.2% | 17.8% | 15.6% | 33.3% |
| Recessed obstacles | 52.2% | 25.6% | 10.0% | 24.4% |
| Mixed indoor | 61.1% | 33.3% | 7.8% | 21.1% |
| All | 55.1% | 27.3% | 10.0% | 24.7% |
Completions are 248/450 versus 123/450. Falls drop from 45.8% to 23.8%. Sparse supports are the hard case. Hardware is a 29-DoF Unitree G1 with onboard depth-to-height-map. The same weights run a one-leg backflip onto blocks, a stair dance, an arm-waving run, a sideways stair walk, and mocap captured on flat ground while the robot steps over randomly placed obstacles. Hardware evidence is qualitative video, not repeated-trial rates.
The usable claim for practitioners: the motion command does not change. Teleop, mocap, and motion libraries still emit a kinematic clip; the policy rewrites footholds from a local height map. That is a different product from a system that picks a parkour skill for you. Identity gating keeps perception as a residual, so a trained tracking prior is less likely to be washed out by the vision branch.
This is incremental systems work. The contribution is scalable offline reference repair plus an explicit command-frame translation in distillation. TCRS does not solve contact dynamics, and it barely replans horizontal footfalls, so hard foothold problems will still fail.
The paper lists three. TCRS is a kinematic synthesizer that assumes a static, rigid, observable height field; deformable, granular, and slippery ground are out of scope. The upper-body command is kept as-is, so arms and torso can strike nearby obstacles. The main ablation table is a TensorBoard window, not a held-out split with seed-level variance.
A few more discounts belong here. Hardware has no success rate. The 55.1% in Table II and the 55.1% in Table IV happen to match; the paper says the denominators are not the same protocol. Sparse-support completion is only 42.2%, far from foundation-model coverage. The authors themselves define BFM as a locomotion-oriented reusable motion interface, not open-ended skill coverage.