SIMPACT uses simulation-in-the-loop to give VLMs physical reasoning with zero training

du_yilun · x · 2026-09-28

A team from Harvard, UIUC, UMD and Amazon FAR (Yilun Du, Jiayuan Mao, Jia-Bin Huang et al.) presents SIMPACT, a CVPR 2026 test-time action planning framework that equips VLMs with physical reasoning via simulation-in-the-loop world modeling — no additional training required.

How it works:

From a single RGB-D observation, it automatically builds a physics simulation: mesh-based for rigid objects, particle-based for deformables (ropes, Play-Doh), with the VLM inferring physical parameters;
The VLM acts as action sampler and optimizer, proposing coarse actions, observing simulated rollouts, and iteratively refining them;
Refined actions are executed in the real world (real2sim2real), letting VLMs do in-context learning from simulation.

The project page includes baseline comparisons and robustness tests against varied reference objects, distractors, and material/shape changes. The authors frame it as extending the real2sim ideas in GPT-Astra demos, turning simulation into a physical reasoning tool for VLMs.

Original post →

SIMPACT uses simulation-in-the-loop to give VLMs physical reasoning with zero training

More from Embodied