Shanghai AI Lab's InfiniHand reconstructs 3D hands from 5,000 hours of egocentric video at 11.19 FPS
量子位 · wechat · 2026-10-14
Shanghai AI Laboratory with Shanghai Jiao Tong University and others propose InfiniHand, a single streaming model that jointly estimates hand position, hand pose and camera trajectory, lifting egocentric hand motions into a shared world coordinate frame to scale up 3D demonstrations for embodied AI.
Method
- A pretrained LingBot-Map backbone extracts scene geometry to localize hands; a hand module fuses local appearance for finger detail, outputs MANO parameters, then maps results to world coordinates via camera trajectory.
- Replaces the fragile detector + pose network + SLAM cascade that compounds jitter, drift and scale errors.
- Streaming windowed processing retains a reference frame, recent features and compressed memory, with background sparse bundle adjustment for long-sequence drift.
Data and training
- 5,000 hours from 9 public datasets (ARCTIC, HOT3D, EgoDex, etc.); Sapiens2-based projection checks removed 30–40% of low-quality candidates.
- Two-stage training: camera-space localization and fine reconstruction first, then camera trajectory and temporal supervision.
Results
- 11.19 FPS prediction throughput, 2.04x HaWoR.
- Lowest PA-p error on all four benchmarks: -21.4% vs. ViDiHand on ARCTIC, -57.7% on EgoDex; world-frame W-MPJPE on EgoDex drops from 78.41mm to 28.77mm (-63.3%) vs. Dyn-HaMR.
- Ego4D natural scenes show robustness to occlusion and out-of-distribution viewpoints.
Paper and code are public (arXiv:2609.35743); after quality screening and embodiment adaptation, the pipeline can feed dexterous-hand data augmentation and imitation learning.
More from Embodied
- iPhone Duo preorders sell out in 3 minutes around NYC, deliveries slip to late November — BenBajarin · 2026-10-16
- UMD's PAIR framework bridges perception and action in VLA models via bridge tokens — Kaixi Feng · 2026-10-16
- CI-MSE offline metric hits -0.87 Spearman correlation with real robot task success vs weak raw MSE — chris_j_paxton · 2026-10-16
- List of 15 best robotics researchers on X, from Jim Fan to Chelsea Finn and Fei-Fei Li — FinanceYF5 · 2026-10-16
- Fake Apple listing for dual-screen 'iPhone Duo' jokes it's the best phone for watching your AI agents — AIFlow_ML · 2026-10-16
- Inside Auki's Hong Kong office: a fake grocery store powering robotics data collection — LoganGrasby · 2026-10-16