AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidance
Shuning Li, Sikai Li, Jiachen Li, Mingyu Ding
cs.RO, cs.AI
2026-06-28
AnyBody learns a spherical latent space any keypoint subset can drive; one wrist still gives 93.7% tracking with coordinated whole-body motion, no mocap suit.
Whole-body humanoid control has been stuck requiring a full motion-capture suit. Physics-based trackers either depend on expensive full-body mocap plus error-prone trajectory retargeting, which bottleneck data collection and policy learning, or they split upper- and lower-body control into separate hierarchical modules and lose the coordinated whole-body motion that loco-manipulation needs. AnyBody wants a controller that takes any subset of body keypoints at deploy time, even a single wrist or just the torso, and synthesizes coordinated whole-body motion.
The core is a single latent motion representation that any keypoint subset can address. Training has three stages.
Stage 1 trains a privileged teacher tracker with PPO on a large unstructured motion corpus (about 140 hours after filtering), then distills it online into a deterministic encoder-decoder whose latent sits on a unit sphere (16-dim, L2-normalized). Forcing the latent onto the sphere removes the norm-drift failure mode of behavior cloning and makes a cosine smoothness regularizer geometrically well-defined.
Stage 2 freezes the decoder and trains a transformer keypoint encoder. Each keypoint becomes a token; masked self-attention lets the encoder accept whichever keypoints are present at deploy time, with missing ones masked via NaN and key-padding. A three-phase masking curriculum goes from fully visible, to annealed partial visibility, to an eight-mode semantic mix (VR, torso-only, single wrist, both ankles, and so on) so the encoder stays robust to arbitrary subsets. The encoder output is aligned to the stage-1 privileged latent by cosine distance.
Stage 3 treats the frozen decoder as a motor prior and runs residual RL in the latent space with PPO. A shallow one-layer transformer predicts a correction to the latent, initialized with gain 0.01 so the correction starts near zero and does not break the prior. This extends the controller to tasks the raw motion corpus does not cover.
Tracking holds up across keypoint configurations:
| Keypoint config | Points | Tracking SR | POI pos. err. |
| Full body (KP5) | 5 | 97.6% | 10.90 cm |
| Upper body | 3 | 94.8% | 9.48 cm |
| Wrists only | 2 | 94.3% | 10.41 cm |
| Single wrist | 1 | 93.7% | 10.79 cm |
| Torso only | 1 | 94.3% | 8.34 cm |
| Ankles only | 2 | 95.4% | 13.39 cm |
For reference, joint-space commanding with privileged information reaches 99.3%. Even one keypoint lands at 93-94%, close to full joint-space commanding.
Downstream tasks after latent-space PPO finetuning all clear 95%:
| Task | No finetune | Finetuned |
| Obstacle-reach open | 54.68% | 97.09% |
| Obstacle-reach barrier | 10.49% | 96.04% |
| Obstacle-reach low-clearance | 44.36% | 99.47% |
| Obstacle-reach container | 3.04% | 95.56% |
| In-air writing | 0.00% | 97.87% |
Container-reaching and in-air writing are nearly impossible without finetuning (3% and 0%), which shows the latent RL is filling in behavior the prior does not cover. On hardware, VR teleoperation with an Apple Vision Pro drives a Unitree G1 using one or two wrist keypoints, and torso-guided 3m forward walking is demonstrated.
For anyone doing humanoid teleoperation or data collection, the direct payoff is dropping the mocap suit and retargeting pipeline. A VR headset plus a few keypoints drives the whole robot, which lowers the data-collection bar a lot. For policy learning, the frozen spherical latent is a reusable motor prior: a new task only needs a shallow residual in latent space, not a full whole-body controller trained from scratch. Swapping the interface from a fixed full-body command to arbitrary sparse keypoints also lets one controller cover settings from single-wrist writing to torso-guided walking.
The authors concede three points. The training corpus deliberately drops crawling, jumping, and extreme locomotion, so the controller destabilizes on out-of-distribution motions; the current platform has no dexterous hand control and cannot do fine grasping; trajectories far outside the corpus fail. All results are in simulation on a single G1 platform, with no cross-hardware generalization shown. The in-air-writing success criterion (within 5 cm throughout) is lenient, and real grasping precision would be stricter.