Grounded Action Model builds robot foundation models on pretrained 3D grounding instead of VLMs
DJiafei · x · 2026-09-22
A new paradigm called Grounded Action Model (GAM) challenges the assumption that language or video models are the right foundation for robotics.
Inspired by developmental psychology (Hespos & Spelke's work on conceptual precursors to language), GAM argues robots should ground objects in 3D first, then learn to act — mirroring how infants grasp before they speak.
Key details:
- Built on a pretrained 3D grounding model (WildDet3D, or SAM3D alternatives) with the backbone frozen; only the action head is trained on robot demonstrations
- Image tokens retain features around selected objects and the robot arm; detection tokens encode point clouds and metric geometry
- An MM-DiT fuses these with robot state history and language to generate action chunks
- Actions can be specified via language, points, or boxes, reducing ambiguity and letting VLMs steer the robot
Paper, code, and project page are released.
Related event: GAM: Grounded Action Model Proposes 3D Grounding as Robot Foundation Base(6 posts)→
More from Embodied
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- Robotics has three regional playbooks: China volume, Europe inspection, US pilots — ingliguori · 2026-11-12
- 3,500 Laser-Armed AI Submarine Robots Patrol Norwegian Salmon Farms — philfung · 2026-11-12
- Model Autonomously Designs a Surgical-Robot-Style Robotic Arm CAD After 75 Minutes — _Stocko_ · 2026-11-12
- Dev releases local world model turning any image into interactive video, runs on RTX 3060 — lucidml_lover · 2026-11-12
- RoboRisk benchmark shows foundation models pose high harm rates in robotics; LEASH harness cuts risk — UoLiverpool · 2026-11-12