Heave-Ho: Decentralized Long-Horizon Multi-Humanoid Collaboration
Ting-Hsuan Chen, Siheng Zhao, Han Jiang, Qi Pan, Louison Lu, Jiaheng Luo, Henghui Bao, Yiyu Chen, Quan Nguyen, Yue Wang
cs.RO, cs.CV, cs.LG
2026-11-17
A frozen humanoid tracker plus a shared residual policy lets robot teams collaborate from local observations and spoken commands; trained on two robots, scales to ten.
A single humanoid cannot move a table or a bed frame. The dominant answer in multi-robot manipulation is a centralized policy that reads team-wide observations and outputs actions for every robot. That design carries two structural costs: input and joint action spaces grow with team size, so a fixed-dimension network needs modification and retraining whenever a robot joins, and gathering team-wide observations at inference requires network communication.
Heave-Ho (USC and VinMotion) goes strictly decentralized. Each robot sees only its own proprioception, a short window of its own reference motion, and the object's dimensions and mass; partners exist only through forces transmitted by the shared object, and not a single network packet is exchanged during a task. The hard part shifts to synchronization: a late lift tilts the object, and a lone step pulls a partner off balance.
Two levels connected by a vocabulary of team skills: lift, put down, steps in each direction, rotations in place.
The low level is residual post-training. A frozen foundation tracking model (SONIC) supplies balance and stepping as the base policy; a residual policy with parameters shared across all robots learns only how to exert force on the object and share the load with partners. Training uses multi-agent PPO: a centralized critic reads all observations during training, while the shared actor acts on local observations alone. It trains in Isaac simulation with 4,096 parallel two-robot environments.
Four choices drive scalability:
Reference motions come from retargeted video of human pairs and procedural inverse kinematics, conditioned on the object's length, width, height, and mass.
The high level runs on a leader robot (or a person). A fine-tuned Qwen3-VL-2B with LoRA reads the leader's head-camera image plus the task instruction and emits one skill token after each skill completes. Synchronization separates what from when: the leader decides what, while a shared coarse grid of control ticks fixes when. The leader speaks the skill token through its speaker, every follower recognizes it with an onboard microphone and schedules it on the next grid slot, and the leader offsets its own start by the expected acoustic delay. Because the grid is coarser than the variation in acoustic delay, small differences in recognition time do not change the start slot. The leader is an assignable role; a person can follow spoken skills or take over as leader.
The accompanying HH-Bench has 12 collaborative tasks for teams of 2 to 10 robots, spanning lifting, shoulder carrying, pulling, pushing, handover, and precise placement; motion capture and inter-robot communication are banned.
Low-level controllers, averaged over the 12 tasks (three seeds of 512 teams each):
| Controller | Sim success rate |
| Pure SONIC (frozen base tracker) | 0.22% |
| Best of remaining baselines/ablations | ≤ 28.56% |
| Heave-Ho (full) | 94.41% |
The full controller's mean body position error is 18.62 cm and object error 19.59 cm. Pure SONIC tracks body motion best yet almost never finishes a task, because the base tracker never learned to push on the object or share a load. Swapping the shared actor for per-robot independent actors drops success to 22.05%.
High-level planners, five long-horizon tasks with 10 trials each:
| Decision maker | Avg success | Latency per decision |
| GPT-6 Astra (zero-shot) | 42% (21/50) | 6.46 s |
| CHORUS (fine-tuned, independent execution) | 12% (6/50) | 78 ms |
| Heave-Ho planner | 86% (43/50) | 52.5 ms |
| Human operator (upper bound) | 98% (49/50) | not reported |
CHORUS runs an independent copy on each robot with no alignment mechanism, so robots start skills at different moments; it fails every task except the briefly coupled box handover (6/10). The Heave-Ho planner takes 52.5 ms per decision on a single RTX 4090 with 5.3 GiB of memory, roughly 120 times faster than GPT-6 through its API.
Real-world experiments on Unitree G1 robots succeed in 92 of 100 trials across all tasks, with stair carry hardest at 7/10 because each stair fits little more than one foot. The two-robot policy deploys without retraining to larger and heterogeneous teams: four G1s carrying a table, G1 + EngineAI PM01 + a person, and similar mixes succeed in 37 of 40 real trials. Ten-robot teams are verified in simulation on the table carry.
Three components are reusable beyond this paper. Parameter sharing plus object-conditioned inputs decouples training scale from deployment scale, sidestepping the retraining tax of centralized approaches. Freezing a foundation model and learning a residual hands the expensive exploration (balance, stepping) to an existing model, leaving RL to learn only the interaction increment. Speech on a shared time grid needs no network stack while letting people join the team, since humans speak and listen natively.
The boundary is equally clear: the skill vocabulary is small, with fixed step lengths and rotation angles. This is a framework for team transport tasks, not general collaborative manipulation.
The paper names two: synchronization depends on followers hearing the leader, which requires a reasonably quiet environment and reliable microphones; and training the planner for a new task needs manually teleoperated demonstrations (24 real episodes per task), which is time-consuming.
Open questions the paper leaves alone: