PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control
Chuhao Chen, Peter Wonka, Chaoyang Wang, Chen Wang, Qiao Feng, Sergey Tulyakov, Lingjie Liu
cs.CV, cs.AI, cs.GR
2026-09-16
PhysStream takes sparse mid-generation velocity kicks plus online scene memory, cutting FVMD about 33% vs the strongest baseline; humans prefer it in over 85% of wild clips.
Controllable video models already take depth, flow, drag tracks, and forces. Most of them still need the full control schedule before the first frame, then synthesize the clip in one shot. A user cannot watch a few generated frames and then decide which object to shove next. Other methods paint pixel trajectories that dictate where objects go. The network copies a path. It does not learn dynamics.
PhysStream, from the University of Pennsylvania and Snap, aims at four properties at once: a sparse signal, a physical quantity, mid-generation edits, and per-object control in a multi-object scene. Concurrent RealWonder touches all four, but routes interaction through an external 3D reconstruction plus a physics engine. That reconstructed state can drift from the pixels, and unmodeled background objects never enter the collision graph.
The backbone is Wan2.2-TI2V-5B. The user supplies sparse velocity increments: at chosen frames, a camera-frame 3D Δv is painted onto the object's first-frame mask, with a neutral value meaning no change. Anchoring to the first frame blocks a leak. Painting velocity at the object's current location lets bidirectional attention peek at future tracks through the condition channels, and the distribution then snaps when training turns causal.
The second condition is structured scene memory, built online. After each latent frame is committed and decoded, Depth-Anything-3 estimates metric depth and back-projects a normalized positional map. SAM2 propagates first-frame masks into a palette-colored tracking map. Both maps come only from already generated frames. They are VAE-encoded, shifted by one latent frame, and concatenated with the noisy latent so causal attention cannot see the current frame's own geometry.
Training is two-stage because three shifts cannot be learned together: velocity control, scene memory, and bidirectional-to-causal attention. Stage 1 finetunes a bidirectional model on velocity increments only. Stage 2 converts it to causal Teacher-Forcing and adds the memory maps. New patch-embedding branches start at zero, so the first step matches the pretrained backbone. The dataset is about 100k indoor tabletop rigid-body clips from SAGE scenes, PyBullet, and Blender, 49 frames at 832×480.
The synthetic benchmark has two 64-clip splits. On the multi-object interactive set, motion-distribution distance FVMD falls from the strongest baseline RealWonder at 1183 to 787, about 33%. Trajectory error Traj-ADE falls from the strongest FlashMotion at 45.67 to 40.24, about 12%. FlashMotion and Tora were given ground-truth center-of-mass tracks for the whole clip; RealWonder skipped reconstruction and used the ground-truth scene. On 20 in-the-wild images, GPT-4o physical commonsense is 4.15 out of 5. Human preference is 91.8% / 88.4% / 91.2% on physics, motion, and visuals.
| Method | FVMD ↓ | Traj-ADE ↓ | Failure ↓ |
| Tora (GT track) | 1463 | 66.79 | 72.00 |
| FlashMotion (GT track) | 3751 | 45.67 | 44.28 |
| RealWonder (GT scene) | 1183 | 60.91 | 64.78 |
| PhysStream | 787.0 | 40.24 | 43.15 |
Ablations: only runs with the tracking map finish the ceramic-dish zig-zag that knocks vases. The positional map cuts Traj-ADE by 15.5% on the 10% of objects with the largest depth travel. Trained on 49 frames, the model is tested to 301. Frames 1–100 respond to 100% of events at 95.2% directional accuracy; after frame 201 appearance drifts, but control remains (87.7% response, 73.7% accuracy). Unaccelerated, 49 frames take 66.3 s on one B6000 (0.74 FPS). Distillation plus a smaller depth model reaches 2.54 FPS.
If interactive control still needs a pre-written trajectory, the workflow is closer to compositing than to play. PhysStream keeps the control on the generated pixels and skips a second simulator that can diverge. For world models and tabletop demos, a sparse velocity kick is easier to specify than dense flow, and closer to a shove.
It is not a real-time physics engine. Unaccelerated throughput stays below 1 FPS, and the validated setting is almost entirely static-camera tabletop rigid bodies.
The authors are direct: extremely complex motion, especially tumbling, still fails. The validated scope is rigid bodies; cloth and bouncing balls need another 10k clips and 5k finetune steps each. Real-time is left for later. The synthetic test set is 64 clips, the human study is 20 images, and several baselines were strengthened. Consistency metrics inflate on nearly static videos, which the paper shows in a figure, so FVMD and track error are the numbers to trust. Teacher-Forcing trains on ground-truth history and infers on its own frames; long-horizon appearance drift is structural.