SIMPACT uses simulation-in-the-loop to give VLMs physical reasoning with zero training
du_yilun · x · 2026-09-28
A team from Harvard, UIUC, UMD and Amazon FAR (Yilun Du, Jiayuan Mao, Jia-Bin Huang et al.) presents SIMPACT, a CVPR 2026 test-time action planning framework that equips VLMs with physical reasoning via simulation-in-the-loop world modeling — no additional training required.
How it works:
- From a single RGB-D observation, it automatically builds a physics simulation: mesh-based for rigid objects, particle-based for deformables (ropes, Play-Doh), with the VLM inferring physical parameters;
- The VLM acts as action sampler and optimizer, proposing coarse actions, observing simulated rollouts, and iteratively refining them;
- Refined actions are executed in the real world (real2sim2real), letting VLMs do in-context learning from simulation.
The project page includes baseline comparisons and robustness tests against varied reference objects, distractors, and material/shape changes. The authors frame it as extending the real2sim ideas in GPT-Astra demos, turning simulation into a physical reasoning tool for VLMs.
More from Embodied
- YC-backed Ornadyne's flapping-wing drone was classified as a bird by Air Force detectors — import_jmr · 2026-11-16
- Uber starts offering Amazon's Zoox robotaxi rides in Las Vegas — Polymarket · 2026-11-16
- Ant Lingbo Embodied AI Challenge crowns TARS team after 48-hour real-robot hackathon — 新智元 · 2026-11-16
- Tesla FSD fleet tops 7 billion city miles, adding ~500M in under four weeks — XFreeze · 2026-11-16
- Ropedia Releases Xperience 2: 1,500+ Scanned Real-World Scenes Paired With Egocentric Demos — liuziwei7 · 2026-11-16
- Hangzhou's Manycore Tech launches Aholo MS07 3D laser scanner with CHCNAV — teortaxesTex · 2026-11-16