RoboTok retrieves web demo videos via 3D hand trajectories, boosting dexterous task success up to 59.8 points
CyberRobooo · x · 2026-09-05
Researchers at Rice University and NVIDIA (Howard Qian, Bowen Wen, Kaiyu Hang, et al.) released RoboTok, an internet-scale data engine for robot manipulation learning, with paper, code, data, and models available.
- Method: given a query human manipulation video, RoboTok organizes web human demonstration videos into a torso-relative 3D hand-trajectory embedding space and retrieves clips with similar hand motions; the representation compares behaviors across camera viewpoints, scene appearance, and occlusions, and stays compact enough for efficient search and continual indexing over internet-scale video.
- Motivation: robot learning needs broad, diverse demonstrations, but collecting robot data is expensive and ill-suited to long-tail tasks — RoboTok makes web video a scalable, continuously growing supervision source.
- Results: RoboTok retrieves more relevant demonstrations and improves downstream policy performance, raising success rates by 17.8, 38.0, and 59.8 percentage points on three challenging dexterous tasks versus the strongest baselines.
Related event: Rice and NVIDIA Unveil RoboTok, a Web-Scale Data Engine for Robots(4 posts)→
More from Embodied
- A classic world model (s, a → s′) acting as a neural MuJoCo simulator — yshan2u · 2026-10-23
- Gen-HumanWholeBody: open 600-hour whole-body manipulation dataset for robots — AdinaYakup · 2026-10-23
- Stanford releases free 520-page robot autonomy course book and lectures — lukas_m_ziegler · 2026-10-23
- Pony.ai and inDrive target Kazakhstan robotaxis by 2027 as Nvidia GPU deployments expand — VectorWireAI · 2026-10-23
- MambaAct cuts VLA robot policy latency up to 59% while lifting success rates — MBZUAI · 2026-10-23
- Founders tour Greater Bay Area: a year of hardware prototyping done in 3 months — DominiqueCAPaul · 2026-10-23