D3GS: Depth, DINO, and RGB Diffusion Co-Guided 3D Gaussian Splatting for Sparse-View Reconstruction
Yunqi Gao, Zhanfeng Liao, Hanzhang Tu, Zhaoqi Su, Guoqing Zheng, Songtao Wang, Hongwen Zhang, Zhou Xue, Leyuan Liu, Yebin Liu
cs.CV
2026-09-19
Tsinghua and collaborators stabilize sparse-view 3DGS with metric depth, DINOv3 features, and single-step RGB diffusion, reaching 17.69 PSNR on 3-view Mip-NeRF 360, 2.40 dB above the next best.
Sparse-view 3D Gaussian Splatting fails in three coupled ways. Geometry is ambiguous because monocular depths lack metric scale, so floaters appear. Cross-view RGB shifts with lighting, and several Gaussians along one ray start fighting for weight. Unobserved regions have no photometric signal, so novel views go blurry or incomplete.
Prior fixes use relative-depth regularizers (FSGS, DNGaussian) or generative cleanup (Difix3D+). Unaligned depth still leaves floaters. Diffusion on a crooked reconstruction paints over the errors. The claim here is that RGB plus depth is not enough; the three failures have to be treated together.
D3GS folds metric depth, DINO structure, and RGB diffusion into one Gaussian optimization. The primitives follow flattened, surface-aligned PGSR Gaussians, each carrying an extra 3-D learnable feature.
Metric depth is estimated coarse-to-fine. MapAnything supplies sparse metric points, then bundle adjustment refines them. After KNN interpolation, a denoising U-Net consumes the sparse depth and a distance-to-anchor map, producing a globally consistent low-resolution metric depth that RANSAC aligns in scale. A DPT decoder fuses that map with multi-scale image features into a high-resolution metric depth. The map initializes Gaussians by back-projection and supervises rasterized depth with L1.
Even accurate depth leaves floaters when colors disagree across views. DINOv3 features are more stable for the same physical point. Features from all training views go through PCA, keeping the top three components as supervision. Gaussians rasterize their 3-D features and match that map with L2. Three dimensions beat 32, because low-frequency structure matters more here.
The diffusion module is not mainly a floater eraser. It is a high-frequency completer. A single-step diffuser is fine-tuned on SynCamMaster: degraded novel-view renders in, reference images as condition, ground-truth as target. Optimization runs T1=3000 photometric steps first, then every T2=100 steps feeds enhanced novel views back into the Gaussians with L1 + LPIPS + SSIM.
LLFF, DTU, and Mip-NeRF 360 are evaluated at 3/6/9 views on PSNR, SSIM, and LPIPS. Training is 15,000 steps on an RTX 3090.
| Data / views | D3GS PSNR | Next best | Gap |
| LLFF 3-view | 21.47 | BinocularGS 21.44 | +0.03 |
| DTU 6-view | 26.04 | CoR-GS 24.51 | +1.53 |
| Mip-NeRF 360 3-view | 17.69 | GenFusion 15.29 | +2.40 |
LLFF 3-view LPIPS falls from BinocularGS 0.168 to 0.131. DTU 6-view LPIPS is cut from 0.068 to 0.034. At 9 views on DTU, CoR-GS wins PSNR by 0.02 dB; D3GS still leads SSIM and LPIPS. On 3-view Mip-NeRF 360, SSIM moves from 0.367 to 0.538 and LPIPS from 0.585 to 0.442.
On a 3-view subset, DTU PSNR is 16.89 with no extras, 19.14 with metric depth, 18.03 with DINO, 17.91 with diffusion, and 20.75 with all three. Swapping in DepthAnythingV2 as the metric source yields 12.62 PSNR on Mip-NeRF 360 against 17.11 for the paper's depth module. One-shot post diffusion trails iterative feedback: 19.34 vs 20.75 PSNR on DTU.
Sparse capture is the default for aerial and quick indoor scans. D3GS splits scale-correct depth, cross-view features, and generative fill into three losses that can be turned off per scene. The metric-depth module is worth far more than dumping DepthAnythingV2 into the same pipeline. Diffusion should wait until geometry is stable, or it will round off structural mistakes.
This is a combinatorial improvement on optimization-based 3DGS, not a new scene representation. It still trains for 15k steps, about 25 minutes per scene, so it is the wrong tool if a feed-forward reconstructor has to return in seconds.
The authors flag the extra cost: depth, DINO, and diffusion add up to about 25 minutes per scene. If geometry is already wrong, diffusion amplifies it; the ablation figures make that visible. Nine-view DTU PSNR does not beat CoR-GS. Details in unseen regions come from a generative prior and can look real while being invented. The diffuser is fine-tuned on SynCamMaster; cross-domain transfer is untested. The protocol stays at 3/6/9 views; two-view and wide-baseline cases are not reported separately.