HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions
Yukang Cao, Haozhe Xie, Fangzhou Hong, Long Zhuo, Zhaoxi Chen, Liang Pan, Ziwei Liu
cs.CV, cs.RO
2026-03-17
HSImul3R rebuilds human-scene contact from sparse views, then a simulator revises motion and geometry: easy Stability-HSI 53.68% vs HSfM 10.52%, penetration 22.9% vs 69.51%.
A reconstruction of someone sitting on a chair can look fine and still be illegal physics: meshes intersect, the chair flies, and the person and object end up standing apart. The paper calls that the perception-simulation gap. Current stacks are split. DUSt3R and Gaussian Splatting rebuild rooms. 4DHumans-style estimators rebuild bodies. Unified systems such as HSfM mostly align in image space and never check whether gravity and contact survive a physics engine.
HSImul3R, from NTU S-Lab, ACE Robotics, and Shanghai AI Laboratory, treats the simulator as a supervisor. It jointly revises human motion and object geometry so sparse-view photos or a monocular video can yield human-scene interaction (HSI) assets that actually simulate, then transfer onto a humanoid.
Reconstruct, then align. Scenes come from DUSt3R. People are tracked with SAM2, lifted to SMPL by 4DHumans, and keyed in 2D by ViTPose, followed by human-scene bundle adjustment. Legs still go missing and surfaces stay non-watertight. Each object is cropped from its clearest view, completed by the image-to-3D model MIDI, then pushed or stuck with contact / non-contact losses so bodies stop tunneling through furniture. The default input is four uncalibrated views.
The simulator then runs two passes.
Monocular video swaps in MegaSAM and TRAM and assumes a static scene with only the person moving. HSIBench films 3 subjects (two male, one female), 19 object types, 50-plus motions, and 300 interactions from 16 synced cameras. Pseudo ground truth is multi-view 2DGS plus SMPL. Refined motions are retargeted to a Unitree G1 with GMR, a diffusion-guided whole-body policy is trained in IsaacGym, and the policy is flashed to the robot over the vendor SDK.
HSIBench is split easy / medium / hard by interaction complexity. Against HSfM, the only sparse-view HSI baseline they compare:
| Method | Easy/Med/Hard Stability-HSI (%) | SP-3D penetration (%) | W-MPJPE / PA-MPJPE |
| HSfM | 10.52 / 4.50 / 2.66 | 69.51 | 5.02 / 2.79 |
| V4 (no DSRO) | 29.56 / 16.62 / 5.17 | n/a | 4.57 / 2.39 |
| HSImul3R | 53.68 / 30.56 / 13.92 | 22.9 | 4.09 / 2.17 |
Stability-HSI requires gravity stability, a settled scene, and remaining contact. Easy-case success moves from 10.52% to 53.68%; penetration falls from 69.51% to 22.9%. Hard cases still sit at 13.92%. Lower W-MPJPE and PA-MPJPE mean the simulated motion stays closer to the pseudo ground truth.
On image-to-3D, fine-tuned MIDI raises gravity stability from 79.19% to 91.50%, cuts Chamfer Distance from 0.198 to 0.173, and lifts F-Score from 81.95 to 88.25. DSO wired to the same MIDI backbone stops at 87.23% gravity stability. Going from 4 to 16 views moves easy Stability-HSI only from 53.68% to 55.16%; extra cameras barely buy penetration or stability. Training is LoRA rank 64, 1800 steps on four A100s. Robot results are qualitative sits and leans on a Unitree G1; no hardware success rate is reported.
For people who build embodied datasets, the paper separates "looks right" from "does not explode in the engine," and it reports penetration and interaction stability as different numbers. Using the simulator as a reward to fine-tune generative geometry is more direct than another contact term on an image loss, and it explains why V4 without DSRO falls back to 29.56% easy-case stability. Four views already saturate the metric, so capture cost can stay sparse.
This is not yet a YouTube-to-simulation factory. Hard-case stability is 13.92%, and the authors say the success rate is not high.
They list three limits: success drops on complex interactions or more than three objects; many failures leave the humanoid and the object standing apart, so the interaction dies in simulation; the tuned MIDI inherits biases from both its original set and HSIBench, which caps out-of-domain use. The monocular branch assumes a static background. Almost all numbers live on the authors' own benchmark; HSfM is the only external method, and V1–V4 are in-house ablations. How "meaningful contact" is decided automatically is thin in the main text. Hardware is qualitative video, with no real-robot analogue of Stability-HSI. Objects are mostly chairs, tables, and sofas, so 19 categories still skew toward sit-and-lean.