InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Video
Kerui Ren, Kaiwen Song, Weiguang Zhao, Yuxi Wang, Yufei Liu, Bo Dai, Haoyu Guo, Chunhua Shen, Mulin Yu, Tao Lu, Junting Dong
cs.CV
2026-09-29
InfiniHand jointly estimates MANO, camera pose, and hand location from ego video; ~5,000h pretraining cuts ARCTIC PA-p 21.4% vs ViDiHand and runs at 11.19 FPS.
Getting usable 3D hand motion out of first-person video means recovering an articulated hand mesh and peeling camera egomotion off that motion at the same time. Current pipelines split the job. HaWoR detects and tracks, runs a camera-space reconstructor, then hands camera trajectory to DROID-SLAM and metric scale to Metric3D. Detection jitter, tracking drift, and scale mismatch travel down the cascade, and the engineering cost is high.
Estimators such as WiLoR and HaWoR also lack large-scale pretraining on unconstrained egocentric video, so messy two-hand interaction and fast head turns still break them. In-the-wild clips almost never come with 3D hand pose and camera trajectory labels, which is the data bottleneck sitting in front of human-to-robot transfer.
InfiniHand puts left/right localization, MANO parameters, and camera trajectory into one streaming feed-forward network. The backbone is LingBot-Map, a streaming 3D foundation model. Geometric Context Attention keeps a persistent memory: anchor tokens fix the reference frame, a recent window holds dense features, and older observations collapse into trajectory tokens.
Training has two stages. Nine public sets (ARCTIC, HOT3D, EgoDex, DexYCB, HO3D, H2O3D, EgoVerse, EgoLive, Xperience-10M) add up to about 5,000 hours, sampled at 10 FPS. Where MANO labels are missing, the model is fit to 3D keypoints. Cleaning is aggressive: Sapiens 2 2D keypoints are matched to projected annotated joints, and sequences over the error cutoff are dropped, removing roughly 30% to 40% of candidates. Stage I trains 300k steps on all valid camera-space hands. Stage II trains 100k steps on the subset with camera trajectories, using 4 anchors plus two 16-frame windows.
Images run at 378x518, hand crops at 256x256. At inference two workers pipeline overlapping windows and a third runs BA in the background when the keyframe pool overflows.
Procrustes-aligned joint error (PA-p) is lowest on all four camera-space benchmarks. On 34 ARCTIC test scenes it falls from ViDiHand's 9.82 mm to 7.72 mm, the 21.4% drop in the abstract. Detection is essentially tied: InfiniHand FAcc/Recall/F1 is 0.993/0.996/0.998 against ViDiHand's 0.997/0.999/0.999. On HOT3D, PA-p drops from 17.22 mm to 7.29 mm. On EgoDex the PA-p gap is thin (13.96 mm to 12.11 mm), and unaligned MP-p gets worse, 30.09 mm to 33.64 mm. Winning the aligned metric does not mean winning every geometric number.
| Setting | Metric | InfiniHand | Baseline |
| ARCTIC camera | PA-p | 7.72 mm | ViDiHand 9.82 mm |
| HOT3D camera | PA-p | 7.29 mm | ViDiHand 17.22 mm |
| ARCTIC world | W-MPJPE | 59.21 mm | WiLoR-SLAM 65.86 mm |
| EgoDex world | W-MPJPE | 28.77 mm | Dyn-HaMR 78.41 mm |
| Speed | FPS | 11.19 | HaWoR 5.48 |
Unaligned world-space W-MPJPE is lowest on all three world benchmarks. Versus WiLoR-SLAM on ARCTIC the cut is 10.1% (65.86 mm to 59.21 mm). Versus Dyn-HaMR on EgoDex it is 63.3% (78.41 mm to 28.77 mm). Sequence-level WA-MPJPE on HOT3D is slightly worse than HaWoR: 35.76 mm against 35.02 mm.
Ablations on ARCTIC isolate the parts. Dropping WiLoR appearance features raises MP-p from 17.09 mm to 33.31 mm. Turning off sparse BA leaves camera-space numbers unchanged and lifts W-MPJPE from 59.21 mm to 80.48 mm. Skipping Stage II joint training sends W-MPJPE to 185.76 mm. Swapping the learned mask head for HaWoR masks lifts PA-p from 7.72 mm to 25.50 mm. WiLoR-SLAM runs at 8.53 FPS and Dyn-HaMR at 0.81 FPS. In-the-wild Ego4D and Xperience results are qualitative only; there is no MANO ground truth.
Embodied training wants world-space traces of how people use their hands, for human-to-robot transfer and for imitation conditioned on retrieved human clips. Cascaded SLAM-plus-hand stacks are slow and brittle. InfiniHand folds that stack into one feed-forward pass plus occasional sparse BA, at 2.04x HaWoR's throughput, with less glue code.
If you already label egocentric video with WiLoR or HaWoR, this is a drop-in candidate; code and a project page are public. It is an incremental system paper, not a new task. The ARCTIC world-space gain over a decent SLAM cascade is about ten percent. The wider gaps sit on EgoDex absolute trajectory, camera-space aligned error, and speed. Scale still depends on an auxiliary alignment model, so metric hand trajectories are not a finished measuring stick.
The paper flags three issues. LingBot-Map has no built-in metric scale, so a post-hoc alignment model can leak error into hand position and global trajectory. High-quality egocentric data with large two-hand motion, fast viewpoint change, and intermittent visibility is still scarce; wild generalization is mostly qualitative. Monocular labels carry time-varying scale drift, so the training targets themselves are not temporally consistent on long sequences.
A few gaps are quieter. 11.19 FPS is the best throughput in the comparison, not 30 FPS real time, and the third worker's BA still costs compute. There is no wild quantitative table. The 10.1% ARCTIC world-space edge is much narrower than the 21.4% camera-space headline. Cleaning throws away 30% to 40% of the data, which makes the remainder cleaner and less like true in-the-wild video. EgoDex camera-space MP-p is worse than ViDiHand, so the SOTA claim lives mainly on aligned error and world-space absolute error.