Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models
Yuanhao Ban, Jiaqi Feng, Hengguang Zhou, Xiaohuan Pei, Justin Cui, Cho-Jui Hsieh
cs.CV, cs.AI
2026-08-20
Stream4D replaces rigid 3D-GS critics with 4D reconstruction plus a motion gate. LongLive 4D-PSNR rises 17.44 to 24.20; humans prefer it to World-R1 on 76% of pairs.
Streaming autoregressive diffusion emits video chunk by chunk, which is how you get long horizons and near-real-time playback. The training loss only scores the next local frames. Geometry, identity, and motion drift as the rollout lengthens. World-R1 and VideoGPA try to fix that with a rigid 3D Gaussian-Splatting critic. Object motion looks like reconstruction error, so the high-reward policy is a frozen scene with camera travel. Autoregressive models lean into the shortcut: once an early chunk stops moving, later chunks copy the same rigid setup and keep collecting reward.
Stream4D swaps the critic for feed-forward 4D Gaussian Splatting. StreamVGGT estimates per-frame cameras; MoVieS reconstructs the candidate as a canonical point cloud plus per-frame deformation and appearance. The reconstruction reward is LPIPS between re-rendered and original frames, clipped to [0,1]. Coherent dynamics score high; identity drift and unexplainable geometry score low. 4D reconstruction does not punish motion, but quieter clips remain slightly easier to reconstruct (Spearman ρ=-0.27), so a motion term is added.
Motion magnitude m is a confidence-weighted 3D scene-flow speed on a dynamic mask. A Gaussian gate g(m) peaks at the base model's median motion, so both freeze-frame and smear lose. A smoothness factor penalizes velocity jumps; a rigidity factor penalizes spatially tearing flow. They multiply: Rmot = g(m)·smooth·rigid. HPSv2 anchors per-frame aesthetics so the geometry terms do not wreck appearance. The three axes are z-scored inside each group and added. Advantages update a LoRA on a frozen distilled backbone through DiffusionNFT's forward-process loss.
Three backbones: Self-Forcing and Causal-Forcing at 5 s (81 frames), LongLive at 10.3 s (165 frames). The test set is 500 motion-heavy VidProM prompts. MoVieS 4D-PSNR: 16.88→20.34 on Self-Forcing, 15.44→20.97 on Causal-Forcing, 17.44→24.20 on LongLive (+6.76 dB). 4DGT, a reconstructor with disjoint architecture, weights, and data, still leads World-R1 by about 0.7 / 1.1 / 2.5 dB. A reward-blind, order-debiased Gemini-3.5-Flash judge gives consistency wins of 82.2% / 73.9% / 74.2% against World-R1's 75.9% / 69.1% / 54.0%. Motion scores are 0.83 / 0.77 / 0.71. VideoReward overall win rates versus the distilled base are 66.2% / 76.0% / 84.4%.
A 50-prompt human study on LongLive (150 pairwise trials) prefers Stream4D overall on 76% of pairs versus World-R1 and 80% versus VideoGPA. Versus the base, raw-motion wins are only 43% (the base moves more, and more sloppily) while overall still sits at 60%. Drop the motion term and PSNR can rise while motion scores collapse to 0.34 / 0.45 / 0.42. Drop reconstruction and motion spikes while consistency falls apart. The gate target mnat=0.020 is the knee: 0.010 freezes, 0.030 smears.
If a geometry reward cannot represent dynamics, the policy will freeze the frame. A 4D reconstructor plus a peaked motion gate separates explainable motion from jitter, and the same recipe transfers across three distilled AR backbones. For world models and interactive video that must keep objects moving over long rollouts, that is more on-target than another static 3D consistency score. It is still reward distillation, not a new generator.
The appendix is blunt: the consistency judge is one vision-LLM, corroborated by VideoReward and the 4D metrics; 4DGT removes MoVieS-specific training bias but still shares a VGGT camera stack. The conclusion flags a streaming 4D reconstructor matched to LongLive's native horizon, and scoring against explicit action or camera inputs. The human study is 50 prompts and five raters. Training uses VidProM for 150 LoRA epochs; a random-500 split lives in the appendix. High reconstruction PSNR is not metric-accurate geometry.