COWBOY: A Scalable Sim-to-Real Framework for Learning Contextual Whole-Body Manipulation
Jiahui Yang, Jason Jingzhou Liu, Ray Muxin Liu, Andrew E. Wang, Kenneth Shaw, Deepak Pathak, Ruslan Salakhutdinov
cs.RO
2026-10-21
A batched whole-body controller reaches, three local RL experts grasp, and one distilled point-cloud policy hits 91% zero-shot success across 24 real scenes.
The same bottle calls for a different stance and a different grasp on an open table, deep on a shelf, or inside a drawer. A mobile manipulator has to pick those choices from a partial onboard view while staying out of the furniture. End-to-end RL can randomize scenes in simulation, but the observation is a high-dimensional layout, the reward has to stay informative across that layout, and the action stacks a base, an arm, a neck, and a dexterous hand. Recent sim-to-real grasping systems change the object and then train in one fixed room.
COWBOY learns scene-conditioned whole-body grasping on a 32-DoF platform: a holonomic TidyBot++ base, a Franka arm, a 16-DoF LEAP Hand, an ARX neck carrying a RealSense D435, and a Unitree L2 LiDAR on the base. Base placement, gaze, arm path, and grasp type are supposed to come from the geometry alone.
Reaching and contact are trained apart. A vectorized whole-body controller (WBC) takes an end-effector pose and a 3D look-at point, and does not receive a base goal. It is built from geometric fabrics: reaching, collision avoidance, gaze, and joint limits are task-space forces, merged by state-dependent metrics into a joint acceleration. The graph is fixed, so it batches on GPU, up to 4096 environments at 60 Hz, about 0.03 ms per environment per step. QP and MPC re-solve a scene-dependent program every tick and do not batch this way. The paper does not claim fabrics are more optimal than optimization-based whole-body control.
Distance splits the work. Far from the target, a base attractor drives the robot in and the arm attractor stays off, so the arm does not stretch across the room. Once the target enters the arm workspace, the base term fades and the arm finishes the reach. Collision spheres on the body push off cuboids, capsules, and spheres.
Contact is three PPO experts in the end-effector frame: top-down into an open container, a side grasp for elongated objects, and a constrained side grasp that extracts an object from a shelf. Observations are privileged simulation state: object bounding box, local walls, hand joints, and fingertips. Actions are relative end-effector commands plus hand joints, tracked by the WBC. Tied to the end-effector frame, one expert can be dropped into thousands of full layouts that keep the same local constraints.
Distillation embeds that local region in random tabletops, drawers, boxes, and shelves, with extra obstacles around it. Rollouts start 0.2–2 m from the object. The WBC approaches until the state matches the expert's reset distribution, which happens on 98.4% of trials; misses still count against the teacher. The expert then grasps. The student is a point-cloud transformer trained with online DAgger. It reads a partial cloud from the neck depth camera and the base LiDAR, proprioception, and a 3D target position, and outputs a 32-D action: base velocity and joint deltas for the arm, hand, and neck. SAM3 seeds that position from a mask centroid. The policy updates it at 60 Hz, and SAM3 corrects at 3 Hz when a detection is valid. Three teacher families collapse into one student on 8192 full scenes and 250 objects, across 8 A6000 GPUs.
The real robot is not finetuned. Mean grasp success is 91% across 24 in-the-wild scenes: tables at different heights, shelves, boxes, chairs, fridges, drawers, and layouts such as stairs and sofas that were never built as real floor plans, only approximated by procedural randomization. Each scene uses 25 objects, a fixed 20 plus 5 drawn from 100 unseen objects, and 3 orientations. Success is a collision-free 10 cm lift. The test objects never appeared in sim training and include irregular and deformable ones. On the same sofa a bottle is taken from the side and a red plush from the top. A blocked base is covered by the arm. Side grasps knock objects over, and nothing recovers them.
Held-out simulation uses 8192 environments and 150 unseen objects. Grasp success (%):
| Method | Table | Drawer | Box | Shelf | Average |
| End-to-end RL | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| ManipGen | 54.8 | 21.2 | 63.1 | 89.2 | 87.2 |
| COWBOY | 94.7 | 90.5 | 92.4 | 92.9 | 91.3 |
End-to-end RL stays at 0% with obstacles after changes to the reward, curriculum, and visual encoder, and reaches 96% once obstacles are removed. ManipGen pairs local point-cloud policies with AIT and a VLM-chosen subgoal. That subgoal ignores the grasp that follows, and the arm often stops near a singularity. COWBOY is higher on all four categories: 90.5% versus 21.2% in drawers, 94.7% versus 54.8% on tables. The reported averages are 91.3% and 87.2%. The expert split is different too. ManipGen trains separate single-object experts. Each COWBOY expert covers hundreds of objects, so far fewer skills are available.
Teacher versus student success is 96.0% / 94.7% on tables, 97.3% / 90.5% in drawers, 95.6% / 92.9% in boxes, and 93.4% / 91.3% on shelves. The drawer gap is the largest. The paper discusses shelves in particular: heavier occlusion, and side extraction that needs tighter avoidance.
Replacing the WBC used by the teacher changes distillation-environment results.
| Controller | Grasp success | Collision | Manipulability | Joint-limit margin |
| Weighted IK | 43.6% | 41.4% | 0.039 | 1.51 rad |
| Fabrics, no avoidance | 74.8% | 32.6% | 0.067 | 1.87 rad |
| Full fabrics WBC | 95.6% | 4.3% | 0.072 | 1.99 rad |
Weighted IK reaches by stretching the arm, so joints sit on their limits. Fabrics without collision avoidance still collide in 32.6% of trials. The full WBC brings collisions down to 4.3% and posts the highest manipulability.
Observation choice decides whether sim scores survive on the robot.
| Observation | Sim success | Real success | Real collision |
| Fixed neck | 78.8% | 46.7% | 8.2% |
| No LiDAR | 75.2% | 71.1% | 31.7% |
| Ground-truth full cloud | 94.4% | 0.0% | 11.3% |
| Rendered depth + LiDAR | 92.4% | 91.1% | 3.9% |
Training on the full cloud scores 94.4% in simulation and 0% on the robot. The rendered-sensor policy lands at 91.1% real success and 3.9% real collision. Locking the neck drops real success to 46.7%, because low objects and box interiors leave the fixed view. Removing LiDAR raises real collisions to 31.7%: the depth camera's field of view and minimum range miss nearby table edges.
At test time there is no whole-body planner and no label for grasp type or base pose. Those choices come out of the point cloud together.
Two design choices matter more than the 91% headline. Do not ask RL to invent a collision-free approach from scratch; the reward can stay on contact. And render the occlusions and field of view the real sensors actually have. In this ablation the full cloud is the best simulation score, and it scores 0% on the robot.
What transfers is grasping. Three local experts cap the behaviors the student can show.
The paper states this limit directly. Grasp type is read off scene and object geometry, while one object can admit several valid grasps once the next task changes. Functional whole-body grasping needs a condition the policy does not take.
Success is a collision-free 10 cm lift. There is no placement and no drawer or fridge interaction. The text says further experts could add skills; those experiments are not here. No mobile-manipulation system is compared on the real robot, so the 0% and 21.2% gaps are simulation gaps.
Which object to grasp arrives as an external 3D position, seeded by SAM3 and corrected at low rate. The policy tracks that point and coordinates the body. Failures caused by a lost or wrong SAM3 mask are not reported separately. Side grasps topple objects into states with no recovery, and shelves are the weaker real setting. Remove active vision or LiDAR and the real numbers move sharply, so the 91% is tied to this 32-DoF sensing setup.