Beyond Pixels: From Video Priors to 4D Worlds
Zihao Liu, Xiaolong Shen, Zhenglin Zhou, Ruijie Quan, Yi Yang
cs.CV
2026-08-11
4D generation either reconstructs geometry from generated RGB video (errors propagate) or binds geometry to one generator (switching needs retraining). Latent-to-4D uses a video model's final denoised VAE latent as a reusable interface, with an L4AR alignment-and-refinement network bridging a pretrained 4D decoder, so one checkpoint transfers unchanged across three video DiTs sharing the same VAE. On Text4D-200 and I4D-200 it beats matched same-latent Wan+4RC cascades in projection DINO-F1 by 2.88 to 3.45 and 5.81 points.
4D generation synthesizes dynamic 3D scenes with explicit geometry and motion that can be viewed from novel viewpoints. The two existing routes each have a hard flaw. Generate-then-reconstruct first synthesizes multi-view or temporal RGB, then lifts it to 4D with a separately trained reconstructor; its RGB interface inserts a reconstructor trained on far narrower data between the video prior and the 4D output, and open-domain content and temporal artifacts propagate into unstable geometry. Integrated feed-forward generation makes geometry a native output and transfers the video prior more directly, but binds 4D prediction to a particular generator or conditioning regime, so swapping in another pretrained video model may require fresh geometry-supervised training. The core challenge, given that 4D supervision is far scarcer than video data, is to build a reusable interface that bypasses generated RGB and lets one geometry-supervised pathway serve multiple compatible video generators.
The key observation is that video generators sharing the same VAE checkpoint, latent normalization, tensor layout, and compression convention produce final denoised latents in a common representation space, even when their DiT backbones and conditioning regimes differ, and this representation sits upstream of RGB decoding. Latent-to-4D uses the video model itself as the encoder, feeding its final denoised VAE latent directly into 4D decoding and bypassing RGB.
The method factors into alignment, refinement, and decoding, centered on the L4AR (Latent-to-4D Alignment and Refinement) network. The alignment module handles the mismatch in temporal resolution, spatial grid, and feature dimension between the VAE latent and the 4D decoder's token grid: a fixed trilinear resampling matches resolution, then a learned 3D convolution aggregates local spatiotemporal neighborhoods and projects channels to the required dimension. The spatiotemporal refinement module adds frame-wise attention (building per-frame spatial structure) alternating with global spatiotemporal attention (exchanging information across all spatial positions and time steps), concatenating multi-level features to give the decoder both intra-frame geometry and cross-frame motion cues. The 4D decoder predicts per-frame, per-pixel depth and world-space rays with confidence, plus a 9DoF camera pose from a camera head, yielding a dynamic point cloud reprojectable across viewpoints and time.
Training encodes about 1,143 annotated reconstruction clips with a frozen VAE and updates only the alignment module, lightweight refinement updates (rank-16 LoRA), and the geometry and camera heads; the video DiT, VAE, original transformer weights, and camera and time tokens stay frozen. At inference the observed-video latent is swapped for a compatible DiT's final denoised latent with the downstream pathway unchanged. As long as DiTs share the same VAE convention, one checkpoint works across them with no condition-specific branch.
On the locked Text4D-200 (text-to-4D) and I4D-200 (image-to-4D) suites, every method runs all 200 cases, rendered from two off-axis cameras. One Latent-to-4D checkpoint transfers unchanged across three Wan-VAE-sharing DiTs. On text-to-4D it beats matched same-latent Wan+4RC cascades in projection DINO-F1 by 2.88 to 3.45 points, and by 5.81 points on image-to-4D, where it ranks first on every metric (above the native 4DNeX). Ablations show clear drops without the 3D convolution or either attention scope. A 50-person multi-view human evaluation prefers Latent-to-4D on geometry fidelity, completeness, temporal stability, and overall quality, with geometry and completeness strongest.
A key diagnostic injects a controlled DiT-derived residual projected onto the alignment null space; at rho=0.6 the point-map drift is 0.0053 and 0.0047 for Latent-to-4D on 7-Scenes and NRGBD versus 0.3827 and 0.3160 for decode-RGB-then-4RC, with the same trend for camera estimation, showing that bypassing RGB truncates the propagation of generation artifacts into geometry.
This loosens the coupling between the generator and the geometry decoder in 4D generation. Using the shared VAE latent space as an interface lets one geometry-supervised pathway serve a family of compatible video generators, so swapping generators or conditioning regimes needs no geometry retraining. For teams working on 4D or 3D generation and world models it is a reusable decoupling point, and it inherits the upstream generator's motion, appearance, trajectory, and manipulation controls without condition-specific training. It also directly addresses the error-propagation problem of generate-then-reconstruct.
The authors are candid. Evidence is limited to the shared-VAE convention; transfer across VAE families is not tested. Projection metrics (off-axis DINO) are appearance-dependent proxies for geometric coherence, not metric 4D accuracy, and the multi-view human study and ground-truth ablations are complementary but also not metric. The controls in Figures 7 and 8 (motion, appearance, pose, trajectory, manipulation, navigation) demonstrate interface compatibility, not action success or physical correctness. Training uses only about 1,143 clips, and 4D supervision scarcity remains a constraint. CogVideoX-5B+4RC retains the highest RGB-reference CLIP-I, and the authors explicitly claim no uniform lead on every metric.