Open-Sourced 3D World Model Predicts Hand-Object Motion From Internet Videos
RoozbehMottaghi · x · 2026-11-17
A UW/UNC/JHU team open-sourced "Grasping the Future," a 3D object-centric world model that takes half a second of human manipulation video and predicts the next 4.5 seconds of wrist trajectories, finger motion, and object movement, aimed at teaching robots manipulation skills.
- Data: 62,562 3D hand-object interactions rebuilt from internet videos (only 12% pass quality checks), plus 8 motion-capture datasets.
- One model, seven tasks: joint forecasting, object given hands, hands given object, goal reaching, and gap filling — all framed as completing hand-object motion, trained once.
- Results: a simulated robot hand executes 73.8% of forecasts along the predicted path.
- Paper, code, and model are open-sourced; data coming soon.
Related event: Grasping the Future: 3D Hand-Object Motion Prediction from Web Videos(3 posts)→
More from Embodied
- Over 200,000 apps now optimized for iPhone Duo, doubling in a week — asymco · 2026-11-17
- Unified hand-object model handles 7 tasks, trained on 100K clips mined from unlabeled video — RoozbehMottaghi · 2026-11-17
- Coco Robotics turns real delivery footage into edge-case training with NVIDIA Cosmos — NVIDIA Developer · 2026-11-17
- NVIDIA's LidarVerse simulates any LiDAR sensor in real time with one world model — NVIDIAAI · 2026-11-17
- Jetson GPIOs now drive $2 L298N motor controllers directly, no Arduino needed — chrismatthieu · 2026-11-17
- UW-led team unveils Grasping the Future, forecasting 3D hand-object motion from internet videos — mangahomanga · 2026-11-17