RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning
Howard Qian, Yiting Chen, Yunfei Xie, Kejia Ren, Podshara Chanrungmaneekul, Gaotian Wang, Bowen Wen, Chen Wei, Kaiyu Hang
cs.CV, cs.RO
2026-09-03
RoboTok matches torso-centric 3D hand trajectories in web video. mAP@20 is 0.35 vs STRAP 0.007; hard lever-sliding PPO hits 79.3% vs HAND 19.5%.
Dexterous robot learning is starved of demonstrations. Teleoperated corpora such as Open X-Embodiment, DROID, and ALOHA are expensive and still miss the long tail of real tasks. Internet video of people using their hands is large and still growing, but looking similar or sharing a semantic label does not mean the hands are doing the same thing. The same bottle-cap motion can show up in unrelated kitchens.
Prior retrieval systems mostly search fixed robot datasets. FlowRetrieval matches optical flow, STRAP aligns visual-foundation-model features with subsequence DTW, and HAND filters by appearance then matches 2D hand or end-effector paths. All three target parallel-jaw grippers, and none take an internet human video as the query. RoboTok, from Rice University and NVIDIA, asks a different question: given a human manipulation clip, pull other web clips whose hand motion matches, then use those clips as guidance for dexterous-hand policies.
Source clips come from Action100M. A filter keeps 4-to-8-second shots with a near-static camera and at most one left and one right hand, then greedily drops overlaps in favor of longer clips. WiLoR estimates 21-joint 3D hands at 5 fps, MoGe-2 lifts the weak-perspective reconstructions into metric camera coordinates, and HaWoR fills missing frames. The result is still camera-framed, so a viewpoint change breaks comparison.
The trajectories are then rewritten in an actor-centric frame. A small torso estimator takes only wrist frames and predicts a static torso frame, so the body itself does not need to be visible. Once canonicalized, Dynamic Time Warping (DTW) becomes the similarity oracle: it aligns two 3D hand-pose sequences through local speed changes and reports a length-normalized negative cost.
Pairwise DTW over 100k clips is too slow to serve as a search index. RoboTok uses that oracle as training signal for a light encoder: per-frame positional encoding, cross-attention pooling, L2-normalized vectors whose inner product should preserve DTW ranking. Training uses 100,000 clips. Each batch of 196 is 49 groups, each group an anchor, two positives from the DTW top-20, and one boundary negative just outside that set. Random negatives are too easy; the boundary case sits on the retrieve-or-not cutoff. A set loss pushes true neighbors above the boundary, a rank loss restores their relative DTW order. At query time every clip is encoded once and search is cosine nearest neighbors. A new clip joins the index with one forward pass.
Retrieval is measured on 10,000 held-out queries against the other 99,999 clips, with the DTW top-20 as the relevant set.
| Method | mAP@20 | Recall@20 | DTW cost@20 (m) |
| Random | 0.000 | near 0 | 4.776 |
| FlowRetrieval | 0.0004 | near chance | 6.612 |
| HAND | 0.0009 | near chance | 4.528 |
| STRAP | 0.007 | 0.12 | 4.044 |
| RoboTok | 0.353 | 0.996 | 1.333 |
| DTW oracle | 1.000 | 1.000 | 1.145 |
RoboTok's mean DTW cost sits 16% above the true neighbors. Kendall τ is 0.487, so the retrieved neighbors are ordered, not merely present. FlowRetrieval is worse than random on DTW cost.
On AssemblyHands (831 two-hand assembly clips with sensor-grade 3D, relevant set k=5) the ranking of methods is unchanged. RoboTok mAP@5 is 0.261 against STRAP 0.133. Mean DTW cost is 1.095 m versus 0.966 m for the true neighbors (13% above) and 1.911 m for random.
Downstream, PPO is trained from scratch in the VTDexManip simulator. Retrieved clips enter only through the reward: a weighted k-NN distance from the current hand state to a bank of retargeted demo poses, plus potential-based shaping that does not change the optimal task policy. No robot demonstrations and no action labels. Eight seeds, 100 attempts per object.
On the original benchmark RoboTok beats the best reported pretrained baseline, VT-JointPretrain, on five of six tasks: +7.45 points on seen objects, +5.83 on unseen. Lever sliding seen is 95.5% vs 89.3%; bottle-cap turning seen is 90.2% vs 83.7%. Table reorientation is the miss: 82.6% vs 85.0%.
That original formulation is saturated. On a harder variant that restores full 3D wrist control and strips dense rewards, the gap opens: bottle-cap turning seen 77.3% vs HAND 59.5%; faucet screwing 44.8% vs 6.8%; lever sliding 79.3% vs 19.5%. Base PPO and random retrieval sit near zero on this split.
For groups training anthropomorphic hands, this is a retrieval path that never collects robot teleop and treats web video as a growing demo library. The query is a clip, not a language command, which fits the case where a target motion already exists and the job is to find more of the same. The encoder is small and the index is additive.
It is not yet a production data engine. The trained index is 100k filtered Action100M clips, not the open web. Downstream tests are simulation only, and the policy sees proprioception plus fingertip force, with no vision. The retrieval ground truth is DTW on the same 3D hand tracks, so in-domain scores partly reflect representation alignment. The AssemblyHands margin is still real, but mAP only about doubles STRAP.
There is no dedicated limitations section. The conclusion flags moving cameras, third-person and egocentric, as future work. The filter already drops moving cameras, multi-hand scenes, and clips outside 4-8 seconds, so the index covers a narrow slice of internet video.
The torso frame is estimated from wrists only, with no ablation of that step. Table reorientation loses to VT-JointPretrain's visuo-tactile pretraining. On unseen bimanual handover, random retrieval scores 42.1% against RoboTok's 34.8%. There is no real-robot experiment. Internet-scale here means 100k-vector search, not a measured pipeline that keeps ingesting new websites.