Grounded Action Model builds robot foundation models on pretrained 3D grounding instead of VLMs
DJiafei · x · 2026-09-22
A new paradigm called Grounded Action Model (GAM) challenges the assumption that language or video models are the right foundation for robotics.
Inspired by developmental psychology (Hespos & Spelke's work on conceptual precursors to language), GAM argues robots should ground objects in 3D first, then learn to act — mirroring how infants grasp before they speak.
Key details:
- Built on a pretrained 3D grounding model (WildDet3D, or SAM3D alternatives) with the backbone frozen; only the action head is trained on robot demonstrations
- Image tokens retain features around selected objects and the robot arm; detection tokens encode point clouds and metric geometry
- An MM-DiT fuses these with robot state history and language to generate action chunks
- Actions can be specified via language, points, or boxes, reducing ambiguity and letting VLMs steer the robot
Paper, code, and project page are released.
Related event: NUS Proposes GAM: 3D Grounding as Foundation for Robot Models(3 posts)→
More from Embodied
- IROS 2026 Hits Pittsburgh: 4,000+ Attendees, Nearly 2,000 Papers — heatherknight · 2026-09-23
- Jim Fan trains robots on human video, calls VR teleop rigs unscalable 'torture devices' — dawnsongtweets · 2026-09-23
- Dev builds an ESP32 face for his AI agent using Copilot CLI, open-sourced — DanWahlin · 2026-09-23
- seeMote Cube debuts: a spatial accessory for Vision Pro with IR 6DoF tracking — OwariDa · 2026-09-22
- Estimating Unitree G1 actuator heat loss in Isaac Sim with a physics-based model — IsaiahBallah · 2026-09-22
- RoboDawn: Tsinghua & Tencent Hunyuan drive robots with frozen VLM, 73.6% one-shot success — _akhaliq · 2026-09-22