Open-Source Guide: Extracting Robot Actions from Egocentric Video
gui_penedo · x · 2026-08-07
Macrodata Labs published an in-depth technical blog on building an RGB-only pipeline to extract metric 3D hand trajectories from egocentric video, providing supervision for robot actions without specialized hardware.
Evaluation & Challenges
The team evaluated full hand-tracking pipelines on the HOT3D dataset using Action MPJPE. A viable system needs >75% direct-prediction coverage and 15 FPS on an H100.
System & Results
Their final system combines WiLoR detection, HaWoR reconstruction, and VGGT-Omega camera trajectory. It achieves 52.04 mm Action MPJPE and 15.53 FPS, reducing error from 59.12 mm and increasing speed from 3.34 FPS compared to the original HaWoR pipeline. Hand reconstruction was identified as the largest error source, with depth accounting for 43% of the action error.
Related event: Open-Source Pipeline Extracts Robot Actions from Egocentric Video(2 posts)→
More from Embodied
- Experiment Showcases 3D Environment and Human Capture Tech — andrew_n_carr · 2026-08-07
- Nucleus Robotics Exits Stealth: Humanoids in German Factories Within 90 Days — lukas_m_ziegler · 2026-08-07
- Humanoid Robots Will Turn Parking lots into Gardens of Eden — davidpattersonx · 2026-08-07
- AgiBot A3 Humanoid Robot Shows Off Surprisingly Fluid Kung Fu Moves — flngr · 2026-08-07
- Persona.AI Demos Gen 1 Humanoid Robot Performing Welding — Distinct-Question-16 · 2026-08-07
- MAD Society Hosts 2-Hour Physical AI Speed Round — Rewkang · 2026-08-07