DualDiff3D: Dual Structure-Appearance Diffusion Priors for Reliability-Enhanced 3D Gaussian Splatting
Qian Wang, Yu Wang, Weiqi Li, Xinhua Cheng, Xiandong Meng, Ronggang Wang, Jian Zhang
cs.CV
2026-09-02
DualDiff repairs novel views with separate structure and appearance branches, then feeds confident pixels back to 3DGS. DL3DV 3-view PSNR rises from 11.36 to 12.56.
Sparse-view 3D Gaussian Splatting fills unseen cameras with artifacts. Follow-up work repairs those renders with a diffusion model and feeds them back. Video diffusion is slow. Image methods such as DIFIX3D+ concatenate the novel view and the reference inside one self-attention stack. Structure should stay with the broken novel view; appearance should follow the reference. One network holding both fights itself, and the farther the reference, the blurrier the fix. On LLFF, DIFIX without a reference even beats DIFIX with one.
Peking University and Pengcheng Laboratory split the repair into DualDiff, then wrap it in DualDiff3D, a reconstruction loop that treats the diffusion output as untrusted.
Two denoising U-Nets start from SD-Turbo, initialized with DIFIX. The structure branch sees the low-quality novel view; the appearance branch sees the reference. Self-attention in the structure branch becomes Structure-Appearance Attention: queries from the structure hidden state, keys and values from the concatenation of both. The input is the render, not Gaussian noise, at a fixed t=200, i.e. one low-noise denoising step. The structure VAE decoder gets zero-initialized skips and a LoRA tune to limit encode-decode loss.
Even without extra training, this attention layout adds about 0.7 dB PSNR on top of DIFIX weights. When trained, the loss is L2, LPIPS, and a VGG Gram term with weight 0.1. DualDiff-s sees only 24-view degraded renders; DualDiff-m mixes 3, 6, 9, and 24 views.
Reconstruction is a Render-Refine-Optimize loop. Sparse views train for 30k steps, then another 30k with a refine every 2k. Novel cameras are interpolated, with slight extrapolation, among existing ones, then filtered by a pixel confidence mask. Confidence mixes variance across three independent repairs, the gap between repair and the raw render, and a depth-based reprojection error onto references. The 3DGS loss is weighted by that mask, and each batch still holds some real images. Three straight validation drops trigger a rollback that discards the latest repairs.
On held-out DL3DV scenes, 3-view repair PSNR is 10.70 for 3DGS, 13.11 for DIFIX, 13.89 for DualDiff-m. At 24 views DualDiff-s reaches 20.57; DIFIX scores 19.20, below the unfixed 3DGS 19.42, so single-net repair can hurt once views are denser.
For reconstruction, DL3DV 3-view PSNR is 12.56 for DualDiff3D, 11.36 for DIFIX3D, 10.70 for 3DGS; at 24 views, 20.49 versus 20.17 and 19.42. LLFF was never used to train the repair net: 3-view PSNR goes 14.11 / 15.21 / 16.34, 9-view DualDiff3D 21.89. On Mip-NeRF360, 3-view 15.70 slightly tops GSFixer 15.61 and DIFIX3D 14.32. At 6 and 9 views the two trade blows: 9-view PSNR 18.45 versus GSFixer 18.63, LPIPS 0.405 versus 0.420.
LLFF 3-view ablation: dumping repaired views straight back yields 15.43; progressive sampling plus confidence weighting reaches 16.34. One DualDiff repair takes 600 ms versus DIFIX 950 ms. A full scene is about 10 min 40 s versus 8 min 52 s, peak VRAM about 12.6 GB versus 9.9 GB.
Novel-view repair has two jobs that should not share one trunk. Dual attention makes that split concrete, and the 0.7 dB training-free bump says the interface was the first bug, retraining the second. The RRO loop treats diffusion pixels as noisy pseudo-labels, which is closer to what 3DGS actually fears: a small geometric lie will drag the Gaussians off.
The gains are incremental. The authors say extreme sparsity still collapses.
Two U-Nets raise VRAM and wall time; the paper is explicit that this is not an online reconstructor. Extreme sparsity still degrades, and absolute 3-view PSNR remains low. DualDiff3D does not dominate GSFixer on every Mip-NeRF360 split. Confidence weights come from a grid search; whether they travel is untested. Initialization is a random point cloud, so numbers are not directly comparable to SfM-started pipelines.