Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
Kerui Ren, Yingxiang Xu, Kaiwen Song, Linning Xu, Bo Dai, Mulin Yu, Tao Lu
cs.CV
2026-09-29
Real2Gym builds physics-checked gyms from videos and reuses skills on robots: 87.5% SR vs 70.8% GPT-6 Astra Direct Mode, ~74.9% fewer tokens, +33.3 points on four Franka tasks.
Self-improvement by trial and error is expensive on a real arm. Resets are slow, failures can damage the robot or the scene, someone has to watch, hardware wears, and restoring objects after a miss is hard. Simulation would let an agent fail in parallel and reset for free, but only if geometry, articulations, and contacts match the demonstration.
Pipelines such as RialTo still scan, repair meshes, model joints, and tune physics by hand. Multimodal models can now emit a Blender scene from video. Direct GPT-6 Astra reconstructions still miss object scale, relative layout, and camera extrinsics. Wrong scale wrecks grasp clearance. A skewed camera breaks the link between the shown motion and what the agent sees. In fine manipulation those errors become penetration, empty grasps, and scenes that will not run under native physics.
Real2Gym, from Shanghai Artificial Intelligence Laboratory, has two stages: build an executable gym from video, then accumulate skills inside it. Inputs are a human or robot demonstration plus the target URDF.
Gym construction has three loops. Geometry is estimated from the first frame with MoGe-3 (monocular) or the Pi3X implementation of π³ (multi-view). SAM2 segments instances across cameras; calibrated cameras and metric cues pin global scale. Occlusions are filled from RGB silhouettes and up to ten sparse frames. The robot is imported from URDF into the same frame, and object poses plus cameras are jointly refined by reprojection. Actions are recovered at keyframes where contact, grasp, support, or containment topology changes. Joint logs are used when present; human hands are mapped onto the target arm. The agent self-checks five diagnostics: main discrepancy, camera alignment, relative placement, contact/penetration, and appearance. The aligned scene is then run in MuJoCo: approach and contact orientation first, then close, hold, support, and release, then a full start-to-end rollout. The physics trajectory is reimported into Blender for Real / Blender / MuJoCo RGB comparison. Validated seed scenes are perturbed one factor at a time (geometry, pose, support height, materials, distractors, background, lighting). Each variant must finish the task under native physics and stay consistent across renderers, or it is sent back.
Skills do not update weights. GPT-6 Astra (medium reasoning) emits a Python program for one subtask, with entry conditions, intermediate checks, and an observable done condition. The program calls SAM3 and GraspNet, then issues pose and gripper commands. One response can chain approach, close, and grasp verification. After the episode a separate extractor writes skills: preconditions, steps, effect checks, object-relative motion rules, and recovery. Motion is anchored on an object frame and re-instantiated from the current observation. Simulation and the physical robot share one perception-and-control API.
Reconstruction is scored on 12 DROID robot scenes and 12 EgoDex egocentric human scenes (easy/medium/hard, four each). GPT-6 Astra with high reasoning grades content alignment, viewpoint alignment, action fidelity, and simulation success on a 0-100 scale.
| Method | DROID viewpoint | DROID sim success | EgoDex sim success |
| GPT-5.6 Sol xhigh | 24.17 | 48.96 | 46.77 |
| GPT-6 Astra Medium | 30.00 | 71.08 | 56.59 |
| Real2Gym | 70.83 | 80.88 | 85.76 |
Viewpoint alignment on DROID is 40.83 points above Astra. EgoDex simulation success is 85.76, 51.5% relative to Astra's 56.59. These four scores are model judgments, not the task-success numbers below.
Policies run in the 24 reconstructed MuJoCo gyms with no privileged simulator state. Direct Mode baselines get the same observations, URDF, and APIs and pick actions step by step. Construction, policy, and extraction all use Astra medium. Budget: 50 model decisions per task, 600 control steps per code chunk.
| Method | DROID SR | EgoDex SR | Tokens (M) | Time (min) |
| GPT-5.6 Sol | 58.33% | 41.67% | 8.57 / 13.63 | 25.72 / 28.73 |
| GPT-6 Astra | 75.00% | 66.67% | 1.42 / 2.56 | 7.81 / 10.31 |
| Ours | 75.00% | 83.33% | 0.64 / 0.66 | 8.60 / 9.09 |
| Ours (w/ skills) | 91.67% | 83.33% | 0.51 / 0.49 | 6.32 / 7.05 |
With skills, mean success is 87.5% against 70.8% for Astra Direct Mode (+16.7 points) at about 74.9% fewer policy-execution tokens. Before skills the mean is already 79.2%, with 67.3% fewer tokens. The baseline decides per action; this agent emits a whole stage. Skills flip D10 and D11 on DROID; D5 still fails. E11 and E12 on EgoDex stay failed.
Physical tests use a 7-DoF Franka Emika Research 3 with a Robotiq 2F-85 and two RealSense cameras, three trials per task. Pick Block into Plate: 100% (3/3) vs 33% (1/3); the block is slightly wider than a horizontal gripper span, so the baseline jams. Hang the Mug: 100% both, 1,302.7 s vs 1,522.7 s total. Cup in Microwave and Close Door: 100% vs 33%. Place Plate on Shelf: 0% zero-shot for both. Across four tasks that is 9/12 vs 5/12, +33.3 points. For the shelf task they reconstruct a twin from an uncalibrated third-person human video. Simulation succeeds on the third iteration (100%), and the evolved skills run at 100% on the real arm.
If a team already drives an arm with a coding agent, the usable path is one demo video plus a URDF, a resettable physics-checked gym, then object-relative skills re-grounded on live observations with no weight update. Stage-level programs cut model round-trips. SAM3 and GraspNet take localization and grasping out of pure visual guessing.
This fills video-to-executable-gym, not a new imitation or RL algorithm. The baseline is the same Astra in Direct Mode. There is no head-to-head on labor hours against RialTo or Agentic Real2Sim. The robot study is four tabletop tasks, three trials each. The fit is narrow: a physics-valid gym from video, with a large model as the policy.
The paper lists three. Wrist and static cameras can misalign, so Pi3X clouds ghost or layer. Stage-level code is cheap but sluggish for tight, locally reactive contact. Hardware is a Franka with a parallel-jaw gripper; no dexterous hand or humanoid.
Measurement choices leave gaps. Reconstruction scores come from Astra, and scene construction also uses Astra medium, so judge and system share a family. Real-robot n=3 per task. The shelf 0% to 100% loop is one task from one human video. EgoDex is human motion retargeted onto dual Frankas, so a high reconstruction score does not mean the human skill became a bimanual policy. Direct Mode decides per action while this agent writes a whole stage, so part of the token gap is program granularity.