HSImul3R makes 3D reconstruction simulation-ready, cutting penetration rate from 69.5% to 22.9%

机器之心 · wechat · 2026-09-20

Daxiao Robotics, NTU S-Lab, and Shanghai AI Lab released HSImul3R (accepted to ECCV 2026), targeting the perception-simulation gap in 3D reconstruction: visually correct scenes often collapse or interpenetrate once placed in a physics simulator.

New criteria: the first simulation-ready human-scene interaction reconstruction framework for uncalibrated sparse views (plus monocular video), adding gravity stability, real contact, and interaction stability as core metrics, with a new HSIBench (19 objects, 300 interaction instances).
Physics-in-the-loop: scene-targeted RL optimizes human motion to truly interact with the scene, while Direct Simulation Reward Optimization uses simulation feedback to fix scene geometry — the simulator shifts from judge to active supervisor.
Results: interaction stability of 53.68%/30.56%/13.92% on Easy/Medium/Hard (vs HSfM's 10.52%/4.50%/2.66%); penetration rate cut from 69.51% to 22.90%.
Sim2Real: optimized motions were retargeted to a Unitree G1, trained into whole-body policies, and deployed on the real robot — suggesting internet human videos could become learnable robot skill assets.

Original post →

HSImul3R makes 3D reconstruction simulation-ready, cutting penetration rate from 69.5% to 22.9%

More from Embodied