Ropedia Xperience 2: A Gold Standard for Scaling Physical AI Environments and Behaviors
Hao Li, Fangzhou Hong, Zhaoxi Chen, Hui En Pang, Kairui Hu, Mingze Sun, Runmao Yao, Xinhao Ji, Xiyuan Zheng, Xuanli Chen, Yalun Dai, Yap Kim Hui, Dingwen Zhang, Yuanyuan Gao, Ziang Cao, Cheng Huang Yean, Jackson Peng, Rui Duan, Yaobo Liu, Zetong Wang, Ziwei Liu
cs.CV
2026-11-13
Collected without robots on site: 1,589 metric 3D scans plus 131 hours of egocentric recordings, each capture reusable as sim environments, robot trajectories, and three benchmarks.
Foundation models can steer robots from raw images, but long-horizon, contact-rich tasks in real spaces break them. The evaluation cited here reports GPT-6 Astra at 22.48% average success on the 42-task RoboDojo benchmark, ahead of 40 public policies yet still under a quarter of the tasks, with zero successes in 11 plug-insertion trials and 9 of 23 navigation-and-search trials. The binding constraint is data supply. A reliable robot policy needs both behavior demonstrations and interactive environments to roll out in, and both are expensive: DROID took 50 operators and 12 months to amass 350 hours of teleoperation. Language models have well-characterized scaling laws; physical AI has not even agreed on what to scale. Ropedia Xperience 2 adds 1,589 metric scene scans and a registration pipeline on top of a prior corpus of 10,000 hours of egocentric clips.
Collection is a two-stage protocol run by one operator, with no robot and no motion-capture rig on site. A handheld LiDAR scanner captures each scene once, producing a metric, gravity-aligned mesh and 3DGS model. The operator then wears HOMIE, a head-mounted recorder with four global-shutter fisheye cameras (two forward, two rear); the forward stereo pair drives SLAM, depth, and hand estimation.
An automated pipeline yields 6-DoF camera trajectories, stereo depth, MANO hand pose (21 joints per hand), SMPL-H body motion from EgoAllo (52 joints; legs are unobserved and inferred from a motion prior), and three-level language captions (task, sub-task, action) from a large multimodal model. Registration is the connective step: SIFT matching with RANSAC and PnP localizes recording frames against the scan, a robust Umeyama fit maps recording to scene, and held-out frame error, gravity consistency, and an ICP check assign PASS/WEAK/FAIL labels. After that, video, trajectories, and hand and body motion share one metric scene frame.
Reuse is the point. Each capture feeds four robot-data routes: real-to-sim conversion into interactive environments (one kitchen decomposes into 217 part-level objects), retargeting of recorded motion onto a wheeled humanoid, live teleoperation inside the simulated scenes with automatic resets, and automated long-horizon trajectory generation that scales with compute alone. Capture cost tracks operators rather than robots, one scan serves many recordings, and the data keeps growing after the site visit ends.
The release holds 1,589 scanned scenes (98.7% of 1,610 graded scans), out of an inventory of 596 locations, 74 tasks, 4,230 recordings, and 131 hours; grades split into Perfect 9.3%, Usable 58.8%, and Reuse 31.9% (partial coverage or reconstruction defects). Three benchmarks share the same 20 scenes, with scan reconstructions as ground truth:
| Method | Metric | Result |
| Boxer (3D detection) | AP25 / AP50 | 24.69 / 6.09 |
| Other 6 detection methods | AP25 | all below 7 |
| Gemini-2.5-Pro (spatial) | overall accuracy | 74.2% |
| HoliWorld-VL-4B | overall accuracy | 67.8% |
| Best other open model | overall accuracy | 60.4% |
| π3 (depth & geometry) | traj. ATE, surface F-score | best across all three densities |
No detector exceeds 7 AP50, and LLaVA-3D and SceneScript score zero: tight 3D boxes in cluttered real scenes remain unsolved. On geometry, DUSt3R and MASt3R run out of memory on dense inputs and CUT3R's pose AUC falls from 0.4103 (sparse) to 0.0625 (dense), so more views do not guarantee better geometry. On spatial questions, every model answers at least 74% of height and relative-position items correctly while none exceeds 34% on distance; the 4B HoliWorld-VL-4B beats all larger open models and GPT-5 High at 65.8%.
The robotics side is single-scene demonstrations. Retargeting covers 10.3 minutes across four recordings in one scene: the base drives 96 m, median fingertip error is 0.40–0.55 cm with 95th percentile under 2 cm, and every frame passes velocity, torque, ZMP, and collision checks. Long-horizon generation runs an eight-subtask task (fetch a mug, microwave it, push open the bedroom door, set it on the nightstand, and so on): 30/30 held-out rollouts complete all subtasks with zero rule violations, each trajectory lasts about 2.2 min and costs 10–16 min of wall-clock compute. In the real-to-sim comparison, a scene built by GPT-6 Astra from images alone shifts object colors, sizes, and positions, while the scan-based scene matches the 3DGS view by view.
Robot-learning teams get three layers of value. The metric scans provide ready ground truth for depth, reconstruction, detection, and spatial reasoning, with three benchmarks included. The real-to-sim route turns one scan of any space into an interactive environment with automatic resets, and trajectory generation scales with compute instead of teleoperator hours, so the marginal cost of demonstrations drops far below real-robot collection. The protocol itself, joint capture of environments and behaviors in a shared frame with continuous reuse, is a credible template for the data layer that physical AI scaling laws will need.
It is a dataset and infrastructure paper, not a capability jump: benchmarks cover 20 scenes, each robotics demo is a single-scene proof of concept, and no sim-to-real policy-training loop is closed.
Author-stated: 87.4% of the release is home interiors; 31.9% of scans carry Reuse grade with partial coverage; release grades include policy overrides (91 scans kept as Usable without a QC result, 33 conflicts resolved to the best grade), so they are not a uniform revalidation; hand and body motion are estimates rather than mocap ground truth; captions are model-generated; one rigid transform cannot fix local SLAM drift within a recording.
Two things also stand out on a close read. Retargeting slows recordings 2× before execution, which shows the trajectories are dynamically feasible but not that they work as training data; human-to-dexterous-hand equivalence is unverified. And the 20 benchmark scenes come from the authors' own collection, so the three benchmarks read more as sanity checks than definitive rankings.