WilLaGS: Latent-Conditional 3D Appearance Fields for Robust Gaussian Splatting In-the-Wild
Yuhao Bai, Qianqiu Tan, Lilong Chen, Huanhuan Lv, Lijun Chen
cs.CV
2026-08-28
WilLaGS learns a β-VAE appearance manifold, a latent-conditioned tri-plane field, and teacher-student transient masks. Sacre Coeur hits 25.84 PSNR, +2.28 dB over AsymGS, at 58 FPS.
Rebuilding a landmark from tourist photos breaks 3D Gaussian Splatting in two coupled ways. Lighting, weather, and camera settings paint the same geometry in different colors. Pedestrians and cars break multi-view consistency and leave ghosts. Vanilla 3DGS assumes a static scene under one lighting; neither holds in unconstrained collections.
Most in-the-wild methods split the two problems. Appearance still follows NeRF-W: one independent embedding per image. GS-W, WildGaussians, and Splatfacto-W move that idea onto Gaussians. Discrete vectors do not track continuous weather, and they cannot place local highlights or cast shadows. Transients are either predicted by an extra U-Net that cannot keep up with 3DGS, or masked by a frozen segmenter such as SAM, which misses scene-specific clutter. WilLaGS treats appearance as a sampleable generative manifold and treats transients as regions that disagree with a scene consensus, then trains both in one loop.
Three pieces, trained together.
Appearance first goes through a β-VAE (β=2, 64-D latent). An encoder maps each image to a Gaussian posterior; a KL term pushes that posterior toward a standard normal. β>1 is a deliberate bottleneck: slightly worse reconstruction is accepted so the latent cannot memorize per-image noise. After training, appearance is a continuous manifold. Codes interpolate. Sampling z from N(0,I) yields weather the training set never showed.
A single global latent still cannot place a local shadow. A hypernetwork reads z and emits three orthogonal feature planes (64×64, 32 channels each). Each Gaussian bilinear-samples those planes at its 3D position, concatenates the hits, and runs a small MLP for a dynamic appearance feature. It also stores an intrinsic feature (albedo-like) that should stay fixed under lighting changes. A color decoder mixes dynamic feature, intrinsic feature, and view direction. Global style enters; local lighting follows position; material tries not to move.
Transients do not use an external segmenter. The student updates every step. The teacher is an exponential moving average of student weights and renders a “scene consensus” for the current view. Ground truth and that pseudo-image go through a frozen VGG. Perceptual L1 on relu12 / relu22 / relu33 becomes a difference map; pixels below 0.85 count as static and enter the reconstruction loss. Pixel error chases illumination; perceptual error tracks structure. The reconstruction term is weighted L1 plus D-SSIM, plus a tiny VAE loss (weight 0.01). Training runs 30k steps on one RTX 3090.
Benchmarks are Photo Tourism (Brandenburg Gate, Trevi Fountain, Sacre Coeur) and NeRF-OSR (europa, lwp, st, stjohann). Test protocol follows NeRF-W: optimize the appearance latent on the left half of each test image, score the right half.
| Method | Sacre Coeur PSNR | Trevi PSNR | Brandenburg PSNR | GPU hours / FPS |
| 3DGS | 17.49 | 17.35 | 19.80 | 0.4 / 105 |
| AsymGS | 23.56 | 23.91 | 28.49 | 5.6 / 47 |
| WildGaussians | 22.71 | 23.70 | 27.36 | 7.8 / 73 |
| WilLaGS | 25.84 | 24.54 | 29.94 | 0.9 / 58 |
Sacre Coeur is 2.28 dB above AsymGS, the strongest concurrent baseline. On NeRF-OSR, europa is 24.38 (GS-W 23.31), lwp 23.36, st 23.43. On stjohann, PSNR 25.55 sits slightly under GS-W’s 25.72, while SSIM / LPIPS are better (0.906 / 0.161 vs 0.894 / 0.276).
Ablations, mean PSNR on Photo Tourism: swap the VAE for per-image embeddings and the score falls to 22.15 (full model 26.77, about 4.62 dB). Drop the tri-plane field: 25.05. Drop the mask: 26.19, with visible ghosts. Extra trainable VRAM versus vanilla 3DGS is about 933 MB, plus 528 MB for frozen VGG.
Fix one z and change the camera: shadows stay geometrically consistent. Interpolate two codes and day turns into night. Sample the prior and the same geometry renders as overcast or sunset without a reference photo.
Per-image embeddings are the default in in-the-wild 3DGS, and this paper shows they overfit. A continuous latent buys interpolation and unconditional synthesis, which discrete codes cannot do, on top of the PSNR gain. 0.9 GPU-hours and 58 FPS undercut WildGaussians’ 7.8 hours by about an order of magnitude, so the quality is actually usable. For a weather-swappable landmark from tourist photos, this is a more complete package than stacking another segmenter. The geometry is still 3DGS; the contribution is appearance parameterization, not a new primitive.
The authors flag dense occlusions and the lack of physical knobs in the latent: there is no “sun azimuth” slider.
A few scores cut the other way. stjohann PSNR does not beat GS-W, so “best everywhere” is false. Test-time left-half latent optimization is not zero-shot appearance transfer. The 0.85 mask threshold and β=2 live in a sensitivity appendix; the main text does not show cross-scene stability. Early EMA consensus is dirty, and there is no burn-in study of whether the mask deletes static detail. The queue entry has no public repo, so reproduction cost is unknown. The datasets are the usual outdoor landmarks; indoor and night-heavy collections are untested.