Real2Gym builds interactive sim environments from videos, lifting robot task success to 87.5%
量子位 · wechat · 2026-10-13
Researchers from Shanghai AI Lab, SJTU, ZJU and others introduce Real2Gym, which turns human or robot manipulation videos into interactive simulation environments where agents can explore, accumulate skills, and transfer them back to real robots. Code and paper are open-sourced.
How it works:
- Multi-view inputs use Pi3X geometric cues plus SAM2 segmentation to constrain object scale/position, building Blender scenes mapped to MuJoCo; errors are corrected around interaction keyframes (grasp, contact, release) and verified by physical execution.
- Each agent decision generates code for a whole manipulation stage, cutting model round-trips; successful and failed experiences are stored in a reusable skill library with model weights frozen.
Results: Across 24 DROID/EgoDex scenarios, Real2Gym beats GPT baselines on reconstruction fidelity and sim success. Reusing first-round experience lifts average success to 87.5% (baseline 70.8%), cuts policy-execution tokens by 74.9%, and drops model responses from 31 to 14. On a real Franka arm, tasks like placing a cup in a microwave all succeed, and a previously failing plate-stacking task is solved after sim iteration.
More from Embodied
- RealSense SDK adds Mac support, web-based viewer and an AI assistant — chrismatthieu · 2026-10-13
- Robots spar hand-to-hand in fluid demo: 'fighting is like dancing' — Puzzleheaded-Mall528 · 2026-10-13
- Spranic Demos Spatial Memory That Turns Building Scans Into Queryable Asset Records — AI Engineer · 2026-10-13
- First look at Apple HomeView shows a chunkier device than expected — talkaboutdesign · 2026-10-13
- Grounded Action Model Bets on 3D Grounding as the Foundation for Robotics, Fully Open-Sourced — joemeno · 2026-10-13
- Podcast: why humanoids get to defense before your home does — audrow · 2026-10-13