Grounded Action Model: 3D Grounding as a Foundation for Robotics
Gehao Zhang, Weikai Huang, Shailesh Shailesh, Yiyan Peng, Jiafei Duan, Ranjay Krishna
cs.RO
2026-09-21
GAM freezes WildDet3D and acts on object-centric geometry. It hits 47.6% on randomized RoboTwin vs 30.4% for Abot-M0, and 17/20 under real visual shift vs 4/20 for π0.5.
Robot foundation models sit on vision-language or video-generation backbones. Neither pretraining objective demands metric answers to where an object is and how large it is. That grounding is left for limited robot demos to learn implicitly. Move the layout, change the background, or rename the target, and a memorized visuomotor map breaks.
Grounded Action Model starts from a promptable 3D detector that pins task objects in metric space, then learns control on that object-centric view. Language, 2D points, and 2D boxes share one observation path.
A Flan-T5 span tagger pulls object phrases from language; points and boxes name objects directly. Frozen WildDet3D returns 2D boxes, metric 3D boxes, and a depth map. Image tokens keep a 16×16 grid only where detections or the arm silhouette overlap; the rest is zeroed. Detection tokens encode a 512-point crop plus center, extent, and rotation. Joint history becomes one state token. A four-stream MM-DiT predicts absolute joint chunks with flow matching. The detector stays frozen while the action head trains.
At deployment, CoTracker3 tracks points inside the first boxes. A Molmo2 planner can retarget via points at subtask boundaries without changing the action head.
On RoboTwin the detector is domain-adapted once on simulator boxes and depth, then frozen. The action head trains per task on 50 clean-scene demos and is tested on unseen randomization.
RoboTwin 2.0 has 50 bimanual tasks, 100 rollouts each in clean and randomized scenes.
| Method | Clean | Randomized | Avg |
| GAM (single-task) | 63.0% | 47.6% | 55.3% |
| Spatial Forcing (co-train) | 77.2% | 26.7% | 52.0% |
| Abot-M0 | 57.4% | 30.4% | 43.9% |
| π0.5 (single-task) | 64.0% | 25.9% | 45.0% |
| FastWAM | 77.8% | 1.9% | 39.9% |
| DP3 | 55.2% | 5.0% | 30.1% |
GAM is not the clean-scene winner. Under randomization it lifts the next best 30.4% to 47.6% and keeps 76% of its clean score. On LIBERO-PRO it averages 0.61 across 16 perturbations versus 0.53 for π0.5, with the biggest gaps when targets move or change: Spatial-Pos 0.60 and Spatial-Task 0.88, where baselines collapse. It trails on appearance/size shifts, down to 0.69 on Goal and 0.50 Obj on LIBERO-10.
On a bimanual YAM both methods hit 19/20 in distribution; under visual shift GAM keeps 17/20 versus 4/20 for π0.5. On a Franka with a Molmo2 planner, step completion is 64.7% in distribution and 49.8% out of distribution, versus 24.0% and 17.1% for π0.5 and MolmoAct2. Ablations on 10 tasks: drop image masking or point cropping and the average falls from 46.8% to 22.3% and 12.8%; detection-only 16.0%, image-only 20.3%.
A robot backbone can be pretrained to put named objects in metric coordinates, not to talk or to dream video. When the scene randomizes or the target changes, the observation follows the detector. Points and boxes let a human or a planner retarget without rewriting the controller. Memory stays in the planner. For cluttered desktops this inductive bias is more on-target than stacking more demos. GAM does not beat co-trained VLAs and WAMs in clean scenes. It wins on perturbations.
Grounding errors flow into actions with no recovery. Object-centric filtering can hide unselected obstacles. The RoboTwin detector saw simulator labels, so some Hard-set gain may be domain adaptation rather than filtering. Grasp quality lags when object size changes. Real-robot comparisons mix interfaces: GAM uses clicks or boxes, π0.5 uses language. Oriented 3D boxes are coarse; the authors flag instance segmentation as a finer alternative.