PixSDS: Why Latent SDS Makes Noisy Pixels
Vsevolod Skorokhodov
cs.CV
2026-08-13
EPFL's Skorokhodov traces latent SDS artifacts to VAE pixel drift, where images accumulate noise along encoder-weakly-constrained directions, and repairs the gradient with a decoded latent lookahead step, cutting BRISQUE from 82.1 to 12.9 in 2D optimization.
Score Distillation Sampling (SDS) uses a pretrained diffusion model as a prior and optimizes rendered images against it, forming the basis of text-to-3D. Modern pipelines run distillation in latent space, and a stubborn class of artifacts shows up: structured color blotches and high-frequency texture noise. Prior explanations mostly blamed the 3D side, including mesh extraction, mipmap sampling, and the renderer itself, and fixes were correspondingly patch-level.
This single-author paper from EPFL runs an elimination-style diagnosis:
The root cause is that the latent-to-pixel inverse mapping of a VAE is underconstrained: many visually different images encode to nearly the same latent. A latent objective only constrains the encoded representation, so the optimized image can drift along pixel directions the encoder barely sees, accumulating noise while the latent stays clean. The appendix formalizes this: for 1D convolution-type objectives, gradient descent can push a low-noise initial point toward a noisier solution
The fix, PixSDS, is light enough to leave the diffusion model, renderer, and SDS objective untouched. Each step computes the standard latent SDS gradient, takes a small lookahead step of size beta in latent space, decodes it to get a clean target image, and adds the direction from the current image toward that target onto the SDS gradient with per-pixel normalization. The lookahead is the key design: pulling the image toward its current decoded latent (beta=0) yields clean but unrealistic results, because it drags the image back to the previous iterate. Only decoding the next step preserves the semantic direction while erasing the noisy one, and the ablations confirm it.
Controlled 2D experiments with stable-diffusion-2-base, 100 MS-COCO captions, one V100:
| Method | FID (lower better) | BRISQUE (lower better) | CLIP-IQA Noisiness |
| SDS | 431.9 | 82.1 | 0.031 |
| VSD | 321.5 | 85.8 | 0.104 |
| 2-step-SDS | 229.8 | 26.4 | 0.479 |
| PixSDS+SGD | 223.0 | 12.9 | 0.590 |
| Direct sampling reference | 190.9 | 12.0 | 0.655 |
PixSDS posts the best FID, BRISQUE, and noisiness among SDS-style methods, with BRISQUE dropping from 82.1 to 12.9, essentially matching the direct-sampling reference at 12.0. In 3D, plugging it into DreamGaussian's second stage and LucidDreamer yields cleaner textures and fewer floating Gaussians without dedicated tuning, and it also works with Stable Diffusion 3.
For text-to-3D practitioners, PixSDS is a plug-in gradient repair on the order of a dozen lines of pseudocode. The diagnosis matters more: it relocates the artifact cause from the renderer to VAE geometry, which means any method that optimizes through the VAE inverse mapping, including latent optimization and inversion-style tasks, shares the same failure mode, and future distillation objectives should treat the VAE kernel space as a first-class concern.
The 3D experiments are qualitative only, with no quantitative metrics or user study, so that section is weaker than the 2D one. The author acknowledges that a "white hair ironman" prompt lost its attribute under LucidDreamer and that beta needs per-pipeline tuning (100x the learning rate for LucidDreamer). The direct-sampling reference still beats every SDS-style method on FID, a reminder that optimization-based generation remains behind direct sampling. The diagnosis also leans on the visual judgment that decoded latents are clean, without a quantitative noise measure.