MotionForesight predicts future 3D motion from existing video models

MotionForesight is a method for predicting future 3D scene flow by repurposing an existing video model rather than training a new large model from scratch. The setup is notable because the model only sees the observed first half of a short interaction video, yet is trained to infer the 3D motion that unfolds afterward; the authors also say it generalizes beyond the training set to home and office videos.

Core method

Each interaction video is split into an "observed" segment and a "future" segment. During training, the model receives only the first half's RGB and geometric information as input. The full video is processed offline to recover object masks, metric geometry, camera motion, and dense 3D tracks, which are then used as pseudo-ground-truth supervision. The future frames themselves serve only as supervision targets and are not directly fed into the model.

Model base and training signal

According to the posts, the method starts from TrackCraft3r, which itself is built on Wan2.1. The authors' key intuition is that a pretrained video model already contains temporal priors about how interactions evolve, so the goal is not to retrain a full model but to adapt it for future-motion prediction with a lightweight LoRA. The training supervision is built by offline processing 40,000 Something-Something-V2 interaction videos.

Generalization and release

The authors say the approach is not limited to Something-Something-V2. They also collected short phone videos in home and office settings and used them to test future-motion prediction, highlighting cross-scene generalization. A paper, project page, and code have been released.

2026-07-18 ~ 2026-07-18 · 6 related posts