MEgoVista: Multi-view Ego-aware Motion Estimation for Metric 4D Hands and Head in the Wild
Jiangong Xiao, Zhihao Zhang, Yifei Dong, Chao Ma, Zhouyi Jin, Zhiwen Hou, Li Liu, Weihuang Chen, Hongbin Sun, Maoqing Yao
Northwestern Polytechnical University / Xi'an Jiaotong University / Maniformer
cs.CV
2026-09-15
MEgoVista uses a tri-fisheye headset and stereo depth to reconstruct metric hand/head motion in the wild, measuring 4.29mm finger-pose error against real motion capture.
Training a manipulation policy from human video needs hand motion labeled in real physical units, not arbitrary coordinates. The only sources for that today are studio motion-capture rigs (ARCTIC, Assembly101) and instrumented headsets like Apple Vision Pro, which EgoDex used to record 820 hours of bimanual tasks. Both share two problems: neither leaves a prepared setup, a capture volume or a fixed tabletop, and neither accuracy claim has been checked against an independent reference, it's taken on faith from the hardware spec sheet. Large in-the-wild corpora like Ego4D scale fine but carry no metric 3D hand labels at all. MEgoVista targets that gap: metric hand supervision collected outside a prepared room, with accuracy actually measured against something else.
MEgoVista is an offline pipeline that turns footage from a custom head-mounted rig, MEgo View, into metric two-hand and head trajectories in one gravity-aligned world frame. The rig carries a central fisheye camera, two side fisheye cameras forming a calibrated stereo pair, and a 500Hz IMU. Three choices set it apart from prior egocentric reconstruction work:
The head-pose stage treats the three rigidly mounted cameras as one generalized camera with frozen calibration. Gyroscope and accelerometer readings nominate keyframes so fast head turns aren't missed, and an incremental mapper in the style of COLMAP builds the trajectory. Scale is injected only once, at initialization: a single stereo pair at one timestamp is triangulated against the known baseline to seed the true metric scale, after which the reconstruction is free to grow and potentially drift. Gravity direction is read from the IMU only after the visual reconstruction is finished and applied as one global rotation, so inertial drift never corrupts the geometry itself.
Validation happened inside a 26-camera, 120Hz Chingmu motion-capture volume, with the headset worn normally and PTP-synced to the reference. A checkerboard reprojection check validated the calibration chain first, with mean errors of 0.62-1.25px static and 0.83-2.51px in motion.
Head trajectory (ATE RMSE, steady-state):
| Method | Session 1 | Session 2 |
| ORB-SLAM3 (offline-refined) | 2.12 mm | 0.92 mm |
| MEgoVista | 2.83 mm | 0.65 mm |
| DROID-W | 26.18 mm (18/22 episodes; 4 failed scale by 0.8-2.68x) | not tested |
| MASt3R-SLAM | 121.59 mm (21/22 episodes; 1 lost tracking) | not tested |
MEgoVista and ORB-SLAM3 trade wins across sessions, both within a few millimeters. DROID-W and MASt3R-SLAM fail outright on a meaningful fraction of episodes, with DROID-W's scale off by up to 2.68x. MEgoVista also runs about 3-3.5x faster per episode (206s vs. 627.5s and 715.4s).
Hand reconstruction against open-source baselines:
| Method | Detection F1 | PA-MPJPE-p (mm) | EPE-P (px) | Jitter (mm/frame²) |
| Dyn-HaMR | 0.84 | 49.18 | 96.42 | 3.50 |
| HaWoR | 0.98 | 16.30 | 83.10 | 16.71 |
| HaPTIC | fails in multi-person scenes | N/A | N/A | N/A |
| MEgoVista | 1.00 | 4.29 | 9.90 | 1.11 |
Finger-pose error (PA-MPJPE-p, which removes global orientation and hand scale) drops to 4.29mm, a 70% reduction from the best baseline, HaWoR's 16.30mm. Absolute position error (EPE-P) falls from 83.10 to 9.90 pixels, an 8.4x gap. HaPTIC breaks down entirely once a bystander's hand enters the scene, which is routine in real footage rather than an edge case.
This offers a path to metric hand-motion labels for embodied AI without renting a studio or buying an Apple Vision Pro: a custom headset worn during ordinary tasks, scored against independent motion capture instead of trusted on spec. The 4.29mm finger-pose error and sub-centimeter head trajectory are strong numbers if they hold up outside this one validation setup. The broader lesson is about where scale has to come from: injecting it at initialization from calibrated hardware, rather than recovering it later through alignment, is what makes DROID-W and MASt3R-SLAM fail on a chunk of episodes and what MEgoVista avoids.
The paper is upfront that its own reference isn't fully independently checked: the checkerboard test validates the rigid transform between motion capture and camera, not the vendor's marker-to-skeleton solver that produces the hand ground truth. A rule-based filter catches gross solver failures (frozen fingers, error spikes, fingertip-marker gaps over 40mm) but not slow drift, and any such residual error gets charged against MEgoVista rather than the reference. Fitted scale deviation also doesn't reliably predict trajectory accuracy, with correlation flipping sign between the two sessions (+0.30 and -0.44). Head-tracking results are mixed against ORB-SLAM3 rather than a clean win. The authors say it's genuinely undecided whether further gains need a better estimator or a better reference.