UniWorld-View: Large-Baseline View Synthesis via Video Diffusion Models
Haiyang Zhou, Wangbo Yu, Chaoran Feng, Xunyu Zhou, Yonghong Tian, Li Yuan
cs.CV
2026-08-05
Large-baseline view synthesis breaks when naive point-cloud rendering tears foreground from background. UniWorld-View fixes this with triple-reprojection and normal filtering, topping WorldScore static and zero-shot NVS.
Novel view synthesis (NVS), generating frames at new camera angles from existing ones, underpins VR/AR, gaming, and social-media immersion. Reconstruction methods (NeRF, 3D Gaussian Splatting or 3DGS) need dense multi-view capture and per-scene optimization; with sparse input their geometry breaks down and occlusions fill with artifacts. Generative methods (diffusion models) loosen the data demands but treat camera pose as an auxiliary condition with no explicit 3D modeling, so camera control is imprecise.
A recent mainstream line lifts monocular input into a point cloud with feed-forward geometry estimators and feeds point-cloud renderings as a geometric condition to a video diffusion model (ViewCrafter, GEN3C, Uni3C, and others). Point clouds give accurate priors, but these methods use naive rendering: under large-baseline shifts, foreground texture gets stretched across the background and background pixels land on the foreground, a failure called foreground-background tearing. That ambiguous geometry then poisons the diffusion output.
The core of UniWorld-View is occlusion-aware point-cloud rendering, a two-step cleanup of tearing and back-face artifacts.
The first step is triple-reprojection disambiguation. The source frames are back-projected into a point cloud and rendered into intermediate views along the target camera trajectory; those intermediate views are back-projected again and re-rendered along the source trajectory to produce a visibility mask that marks which source pixels stay valid under the target trajectory. Accumulating these masks over time removes foreground-background tearing.
The second step is normal-based back-face culling. Points on surfaces facing away from the target camera get projected by mistake and add geometric ambiguity. UniWorld-View estimates each point's normal and keeps only points whose normal aligns closely enough with the viewing direction (normal dot view above cos alpha), producing a front-facing mask. The final render applies both the visibility and front-facing masks.
The render feeds a dual-stream conditional video diffusion model built on VACE (fine-tuned from WAN2.1-14B). The two conditions divide the work. The point-cloud render plus validity mask (spatially aligned but geometrically incomplete) goes through the Context Blocks to anchor the 3D structure and enforce the target trajectory. The source video (visually complete but spatially misaligned) goes through new Ref-DiT blocks that use cross-attention (novel-view features as queries, source video as keys and values) to pull in source appearance, filling occluded regions and repairing texture degraded by the point cloud.
Training runs in two stages, 10K iterations each on 32 GPUs. Stage one trains only the Context Blocks on 100K self-supervised dynamic monocular pairs (OpenVid-1M, depth from VideoDepthAnything) with the DiT and reference branch frozen, forcing the model to handle imperfect point clouds. Stage two trains the Ref-DiT blocks on 100K static multi-view triplets (DL3DV, RealEstate10K, point clouds and poses from VGGT) to learn texture completion. Inputs are 480x832, 81 frames.
On the WorldScore benchmark, UniWorld-View ranks first among the eight leaders with a static score of 85.53 (above WorldScape-0.2 at 85.13 and World Dreamer at 84.52). It takes first on all three controllability metrics (camera control 97.72, object control 88.98, content alignment 86.61) and on photometric consistency (94.11). Its dynamic score, 76.09, is second, behind only WorldScape-0.2 at 76.23.
Zero-shot NVS on RealEstate10K, CO3D, and DL3DV against See3D, GEN3C, Uni3C, and SEVA:
| Dataset | Metric | UniWorld-View | Runner-up |
| RealEstate10K | PSNR | 21.73 | 21.54 (Uni3C) |
| CO3D | PSNR | 20.00 | 19.11 (GEN3C) |
| DL3DV | PSNR | 15.82 | 15.41 (Uni3C) |
It posts the best PSNR and SSIM on all three datasets, the best LPIPS on CO3D, and the second-best LPIPS on DL3DV.
This work fills a neglected gap on the point-cloud-as-condition route: the quality of the rendering itself. Prior work assumed point-cloud rendering was good enough and concentrated on the diffusion side; UniWorld-View shows that tearing and back-face projection under large baselines are the real ceiling, and deterministic geometry fixes (triple reprojection, normal filtering) lift results markedly. For practitioners it is a reusable conditioning preprocessor: any pipeline that guides a video diffusion model with point-cloud renderings (ViewCrafter, GEN3C, and their kin) can swap in this occlusion-aware renderer. The work also closes the loop on monocular 4D reconstruction, first synthesizing a set of spatio-temporally consistent multi-view videos and then feeding them to a 4D Gaussian optimization, sidestepping the hard constraint of multi-view capture.
The paper has no dedicated limitations section; what follows is partly from the authors' framing and partly from reading. On WorldScore, style consistency (76.55) is UniWorld-View's weakest metric and sits low among the eight leaders, so matching the style of the prompt or reference is not a strength. Its dynamic score is only second, achieved with a two-stage pipeline that generates a reference video and then conditions the evaluated video on it, a procedure that introduces its own error. On zero-shot NVS, RealEstate10K LPIPS loses to GEN3C and Uni3C, a trade-off the authors concede in favor of pixel fidelity and structural similarity. The full 4D pipeline leans on a long chain of off-the-shelf models (SAM2, video inpainting, Stream3R, VideoDepthAnything, VGGT), so engineering complexity is high and the quality of monocular depth and pose estimation remains a hidden ceiling.