VPM: teaching video world models to forecast 3D point trajectories improves videos and robots
sainingxie · x · 2026-10-21
Researchers from KAIST, ETH Zürich, SNU, NYU, RLWRLD and Physics Sim Lab (advised by Saining Xie and Marc Pollefeys) introduce VPM (Video-Point Model), a 4D world model that jointly forecasts future videos from arbitrary viewpoints and persistent 3D point trajectories, given posed multi-view histories and a language instruction.
Motivation: multi-view world models can render each camera convincingly while disagreeing about how the scene moves in 3D — per-view realism doesn't guarantee coherent 3D motion. VPM uses persistent point trajectories as shared supervision tying appearance changes across all views to one 3D motion description.
Findings: explicitly forecasting point tracks makes videos more faithful and 3D-consistent, and the same point supervision improves robot action learning; predicted tracks enable zero-shot control. The project also releases a simulated+real dataset (VPM-Data) and benchmark (VP-WorldBench).
More from Embodied
- The Verge editor relieved he skipped the $2,000 iPhone Duo after scathing review — tomwarren · 2026-10-21
- PhysCanvas world model predicts vision and touch, lifting sim-real correlation to 0.89 — stepjamUK · 2026-10-21
- CoreWeave launches Physical AI Launchables: run robotics and simulation workloads with one command — ayushthakur0 · 2026-10-21
- David Patterson: The ChatGPT Moment for Robots Will Come From ChatGPT Itself — davidpattersonx · 2026-10-21
- SCAMPER: a virtual rat with realistic motion for RL-based neuroscience experiments — tyrell_turing · 2026-10-21
- iPhone Duo week-long review: fully redesigned dual-screen apps make or break the experience — BenBajarin · 2026-10-21