Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning
Zishuo Li, Bowen Yang, Changtao Miao, Kai Zhu, Hao Chen, Qingze Guan, Zhengxing Wu, Wanke Zhan, Yang Sun, Zhiyi Huang, Zitong Shan, Zhenchao Jin, Jiadong Hong, Taowen Wang, Yushi Feng, You Liu, Yibo Wang, Yifan Yang, Zhaowen Zhou, Man Luo, Hao Cheng, Bo Zhang, Jianshu Li, Jiansheng Cai, Guocai Yao, Jize Zhang, Chenhao Lin, Renjing Xu, Lequan Yu, Chao Shen, Chunhua Shen, Zhe Li
cs.RO, cs.CV
2026-07-15
Ant Research collected about 2,000 hours of egocentric manipulation video from 500-plus contributors using 400-plus phones, annotated with MANO hand poses, camera trajectories, and atomic actions, plus a reusable processing and training toolchain.
Embodied foundation models are bottlenecked by the scale and quality of real interaction data. Existing egocentric datasets miss in two directions. Those that use specialized head-mounted hardware, like EgoDex and EgoLive, are hard to scale because of the equipment. Large-scale passive video, like Ego4D and EPIC-KITCHENS, lacks the hand motion, camera trajectories, and action boundaries that robot training needs. The paper draws a sharp line: releasing video is not the same as releasing data that can be used directly for training.
Open-AoE's first release is about 2,000 hours of manipulation video, collected by more than 500 contributors using over 400 phone models in natural environments, covering more than 400 scenes and 8,000 tasks. Annotations include English text, MANO-based hand poses, camera trajectories, and temporally aligned atomic actions.
Turning raw video into trainable samples is a four-stage pipeline. The first stage is on-device detection, where lightweight models gate recording on hand visibility and wearing compliance. The second does offline quality control: slicing, frame-rate fixing, and VLM-based semantic labeling. The third handles reconstruction and annotation, estimating camera trajectories with DROID-W, reconstructing MANO hand meshes with HaWoR, and segmenting atomic actions. The fourth is a three-gate quality check for completeness, correctness, and consistency. Phone-based capture keeps cost down; this pipeline turns it into structured data.
The downstream side is a full toolchain. A visualization tool overlays MANO meshes, keypoints, and trajectories. A retargeting tool supports cross-embodiment transfer (G1, Galbot/Galaxea, Sharpa, XHand) and generates paired human-robot video for domain transfer. A training interface provides model-specific action representations: 110D dense MANO for LeRobot and world-action models, 48D wrist-fingertip for H-RDT, 62D Sharpa for GR00T N1.7, and 20D gripper for SmolVLA, with recipes for VLA policies, world-action models (WAMs), and world models.
From a 100-hour sample, the paper reports 32,407 distinct natural-language action descriptions, 175 verbs, 8,030 objects, and 135 scenes, with 99.99 percent temporal coverage and 13.97 segments per minute.
| Metric | Open-AoE | Comparison |
| Temporal coverage | 99.99% | OpenEgo 50.1% |
| Image-annotation consistency | 4.583 / 5 | OpenEgo 3.029, EgoDex 2.916 |
| Effective Rank (visual diversity) | 97.43 | ranks first in most trials |
It is the only dataset where all five supervision modalities (action, bounding boxes, confidence, hand pose, camera pose) are universally available.
For anyone working on human-to-robot transfer, VLA training, or world models, this is open infrastructure you can pick up and use: cheap capture, structured annotations, and a toolchain that covers everything from retargeting to training. It lowers the barrier on both contributing data and reusing it.
The authors draw their own boundaries. All statistics come from a random 100-hour sample, not the full set. The downstream benefit of camera-domain diversity is still a training hypothesis that needs controlled ablations. The consistency scores against other datasets are a cross-dataset audit under a shared evaluator, not a controlled benchmark on a shared ontology; the semantic-breadth comparison measures the breadth of released semantic supervision, again not a controlled comparison under a shared action ontology. These comparisons show that Open-AoE is more completely and densely annotated, not that it is the most accurate under one uniform yardstick.