Rice & NVIDIA unveil RoboTok: 3D hand-trajectory retrieval mines web video for robot training data

CyberRobooo · x · 2026-09-05

Rice University and NVIDIA introduce RoboTok, an internet-scale data engine for robot manipulation learning — think a "TikTok for humanoid robots."

Problem: robot manipulation data is scarce and expensive to collect, poorly covering the long tail of real-world tasks, so the team mines useful human demonstrations from massive web video.
Method: instead of caption- or object-based search, RoboTok embeds videos in a torso-relative 3D hand-trajectory latent space and retrieves clips with similar hand motions, robust to viewpoint, scene, and occlusion changes, and supports continual indexing at internet scale.
Results: on challenging dexterous tasks like bottle-cap twisting, faucet turning, and bimanual handovers, RoboTok lifted success rates by 17.8, 38.0, and 59.8 percentage points over the strongest baselines.
Significance: establishes hand-pose trajectory-aware retrieval as a path to making web video a scalable, ever-growing supervision source for robot learning — though how much of this translates into real humanoid capability remains an open benchmark.

More from Embodied