Physically Based Rendering in the Latent Space
Vuk Radovanovic, Vishesh Gupta, Adrien Gruson, Binh-Son Hua
cs.GR, cs.AI, cs.LG
2026-09-18
Trinity College Dublin path-traces signed radiance into an SD3.5 VAE, then a per-scene refiner cleans residuals. Cornell Box camera edits drop LPIPS from 0.502 to 0.075.
Diffusion models generate well and control poorly. Physically based rendering controls well and will not invent photographic texture statistics on its own. The two meet in a VAE latent space: Stable Diffusion 3.5 maps a 1024×1024×3 image to 128×128×16, about 12× compression, and those channels still show edges, shadows, and emitters.
Path-tracing in RGB and then encoding means every scene edit pays for the encoder. If light transport can emit latents directly, scene parameters can talk to a diffusion prior without that hop.
Latents are not a linear RGB code. Emitters are signed, occluded regions can out-intensity lit ones, and some channels look almost flat. The classical rendering equation does not produce those effects.
The method splits latents into a physically renderable part and a residual. The physical part rewrites the rendering integral as three terms:
All three are estimated along one path. Positive and negative values are gamma-corrected separately at γ=2.2. Mitsuba path-replay backpropagation then fits per-vertex signed emission, flat response, occlusion, and BSDF parameters with a Huber loss (δ=1) in latent space for 3,000 steps.
A small per-scene convolutional MLP predicts the residual from the rendered latent plus normal and depth AOVs. Its loss is latent MSE+SSIM plus LPIPS on the decoded RGB, trained for another 3,000 steps. On a Cornell Box, each stage takes about five minutes on an RTX 4090. Training uses a single encoded reference rendering.
The baseline renders 16 non-negative channels with the classical integral, logit-maps them into the native latent range, and trains the same refiner. On the Lamp scene, the rewritten equation cuts color shifts and leftover artifacts.
Ablations report mean LPIPS / MSE×100 on decoded RGB. Cornell Box camera motion: signed emission only 0.502 / 2.264; plus flat response 0.228 / 0.335; plus occlusion 0.213 / 0.222; after the refiner 0.075 / 0.069. Object motion reaches 0.032 LPIPS; light motion 0.013. Lamp and Living Room follow the same pattern; large light moves are the hardest.
On the training view the refiner nearly matches the reference. After camera, light, or object edits, the physical term still carries content, shadows, and highlights; the refiner sharpens edges and contrast. Monte Carlo estimates still converge with spp in latent space; the refiner and decoder add bias.
Runtime depends on the target. For latents, the method is 42–84× faster than RGB path tracing plus encoding. For RGB pixels, one decoder eval dominates and the method is 6–35× slower than RGB path tracing at equal error.
This is a first working bridge from light transport into a diffusion VAE, not another ControlNet. After a one-image fit, camera, lights, and objects move like a graphics scene. Score distillation can hit the refined latents and skip the encoder.
It does not replace an RGB renderer. Final pixels are decoder-bound. The refiner overfits one scene and does not transfer.
The authors flag three issues: low latent spatial resolution aliases fine geometry; the refiner is per-scene and single-image; RGB output loses the equal-error runtime contest. Optimization covers only interior derivatives, so visibility discontinuities are out of scope. Experiments assume known geometry, material types, emitter locations, and camera. Textured novel views are still early.