CrossBFM: Distilling a Shared Latent Behavior Space Across Humanoid Embodiments
Tan-Dzung Do, Tuan Dat Phuong, Nico Bohlinger, Cuc T. Trinh, Siwei Ju, Vien Anh Ngo, Jan Peters, Xinchao Wang, An T. Le
cs.RO
2026-09-30
CrossBFM distills G1's BFM latent onto three humanoids in <1 GPU-hour. Tracking loses 0.025 rad vs joint refs; an unseen similar robot recovers 89% of tracking.
Humanoid whole-body control has two command interfaces. One is a joint-space reference: every frame names a target angle, which is how TWIST2-style trackers are trained. The other is a Behavior Foundation Model (BFM). Forward-Backward representations (an unsupervised factorization of where a policy will go) learn a latent space Z, and a single vector z can mean track this clip, reach this pose, or maximize this reward, with policy π(·|z) left frozen.
BFM-Zero brought that interface onto real robots. UFO cut FB pretraining to a bit over 100 hours on a consumer GPU. Each run still serves one body. A second robot means another unsupervised RL job, and the new coordinates do not line up with the old ones. Same geometry, different behaviors at a given point. Every downstream prompt is tied to that space, so a new body doubles the bill and splits the prompt language. Prior cross-embodiment work usually trains one policy on many morphologies, but on a single task such as velocity tracking. The shared object is an embodiment encoder, not a latent that turns a reward into z in closed form.
CrossBFM treats the latent as the thing that should transfer. A Unitree G1 BFM is frozen; only its backward map BS is kept. LAFAN clips are retargeted onto each target robot on a shared timeline, so source and target frames depict the same motion. Correspondence is then a retargeting fact, and the target problem is supervised: regress new proprioception onto the source latent z★.
The encoder has no robot-specific weights. Every body is packed into a 163-dim canonical view: eight key bodies (left/right knee, foot, elbow, arm tip), pelvis height/orientation/linear velocity, and 33 joint slots. Missing DoF are zero-padded with a presence mask, so at-zero and does-not-exist are distinguishable. Lengths are divided by that robot's median root height. A 64-frame window (2.13 s at 30 fps) goes through four pre-LayerNorm transformer layers and is radially projected onto the sphere of radius √256, matching the source manifold. Loss is per-frame cosine. 3.27M parameters, under one hour on a consumer GPU.
Post-LayerNorm collapsed to a constant vector that cosine cannot unstick, which is why pre-LN is required. Bidirectional attention peeks about a second into the future and is fine offline. The causal mask is for live control: validation cosine drops from 0.8695 to 0.8420, closed-loop joint MAE from 0.2068 to 0.2043 rad, matching oracle conditioning.
Stage 2 trains a PPO tracker per robot. The actor sees [z | proprio] and no joint reference; the critic keeps privileged future reference in sim and is dropped at deploy. About ten hours per body on an RTX 4090. Reward prompting needs a state distribution ρ that the target never collected in unsupervised RL, so the retargeted corpus is used as a cache: encode, score key-body features, softmax-weighted average, project. Absolute height thresholds are transported by quantile: G1's 1.0 m medium arm raise becomes 0.820 m on T1 and 1.157 m on M3.
A rectified-flow model then samples 64-step latent chunks in the shared Z from a behavior mode, 16 frames of history, and an optional frame-aligned motion condition. Twenty denoising steps, radial projection after each, fed straight into the frozen trackers. No reference motion and no backward map at inference.
Targets: in-house M3 (27 DoF), Booster T1 (23), Fourier N1 (23). All three BFM prompt modes transfer.
| Robot | Joint MAE | Latent MAE | Gap | Goal MAE |
| M3 | 0.1777 rad | 0.2024 | +0.0247 | 0.2345 |
| T1 | 0.1844 | 0.1901 | +0.0057 | 0.1967 |
| N1 | 0.1362 | 0.1391 | +0.0029 | 0.1512 |
The joint-conditioned tracker, trained with the same PPO budget in the same env (TWIST2 setup), is the upper bound. The 0.025 rad gap quoted up front is M3, the worst of the three. Joint policies cannot interpolate between poses; a discontinuous jump in joint targets produces large torques and early termination. Latent interpolation reaches goals with no falls, at most +0.032 rad over tracking.
All 41 reward prompts run. Computing zrew on the source with the true BS yields mean normalized return 0.183 on the targets; computing it on the target cache with quantile thresholds yields 0.431, 2.35× higher. States that are optimal for G1 are often unreachable on the other bodies.
Shrinking the distillation corpus to 25% raises M3 closed-loop joint MAE from 0.2132 to 0.2241, a 5% hit. At 10% it jumps to 0.2706; random latents sit at 0.4411. Unseen clips of seen behavior types cost 1.4 to 6.7 mm extra key-body error.
Leave-one-robot-out: an encoder never shown M3 closes 89.3% of the tracking signal gap (all-three encoder versus random z). N1 closes 55.5%. T1, shorter and morphologically farther, closes 12.8%. Cosine to z★ is only 0.625 even on M3, but the tracker still executes the right cluster.
Real M3: tracking 0.1985 rad (joint) / 0.2206 (latent) versus sim 0.1777 / 0.2024; goal 0.2675 versus 0.2345; reward 0.562 versus 0.508. A unified encoder is slightly worse at matching z★ than three separate encoders (M3 cosine 0.8899 versus 0.9026) and much more consistent across robots (pairwise agreement 0.938–0.965 versus 0.867–0.913).
The expensive part of a BFM is unsupervised RL for the backward map. Once retargeting supplies frame correspondence, that map is an hour of cosine regression. A new body does not need another 10² GPU-hour loop of discriminator, replay, and domain randomization. If a lab already has one trained BFM and a retargeter, the marginal cost is a ten-hour PPO tracker, not a new foundation model.
One z can command several robots. The flow generator can skip the reference motion and emit a behavior-mode stream. That is a more operable multi-body interface than a fresh, unaligned latent per robot.
The step is still incremental. Trackers remain per-body PPO. Zero-shot robot transfer only holds for close morphologies. Reward features drop joint-configuration terms.
The conclusion itself names two scaling axes: richer motion for a finer behavior space, and a larger robot set for unseen bodies. The T1 fold already shows the second: with no close relative in training, only 12.8% of the signal gap closes and cosine falls to 0.25. Interpolate among similar morphologies does not extend to arbitrary humanoids on three robots.
Distillation assumes retargeting is a correspondence oracle. Appendix noise at GMR-scale systematic bias (×1 = 0.027 rad rms) is tolerated; ×8 breaks alignment and tracking together. Per-clip random time shifts have no global phase to search, and the encoder fails to learn them. Retarget quality is a silent dependency. The main tables do not compare two independent retargeters.
Flow-sampled latents are shown as walk, run, and dance rollouts, without a tracking MAE table matching the reference-conditioned numbers. Hardware metrics are a single M3 table. The source BFM is always G1; distilling from a smaller body onto a larger one, or swapping the source model, is untested.
Each robot still trains its own tracker. What is shared is z, not π. A cross-embodiment policy is left for later.