Sparse auto-regressive modeling for scene generation from multi-view images
Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel, Wonjune Cho, Bardienus Pieter Duisterhof, Vincent Leroy, Jerome Revaud
ECCV) 2026
cs.CV, cs.LG
2026-09-03
SPAR3S completes scenes in a sparse voxel latent via masked autoregression, with no 3D ground truth. Two-view 224x224 FID on 3DFront falls from LatentSplat's 180 to 59.
Completing a 3D scene from sparse, unconstrained views means inventing geometry and appearance that the cameras never saw, without a cubic voxel bill. Feed-forward reconstruction (DUSt3R, pixel-aligned 3DGS) only covers what is visible. Generating extra views with multi-view diffusion and lifting them into 3D needs novel cameras up front and still drifts across views. Dense volumetric generators scale poorly, and curated 3D ground truth is scarce.
SPAR3S, from NAVER LABS Europe, folds the problem into a sparse, voxel-aligned 3D latent space. Only occupied voxels are materialized. An encoder-decoder is trained with photometric losses through differentiable 3D Gaussian Splatting, so no 3D mesh or splat ground truth is required.
The encoder takes any number of views. Pointmaps drop empty voxels. 2D patches and 3D voxel tokens update each other with bidirectional cross-attention, mixed with geometry-guided logits from pointmap projections. All information must pass through one voxel-aligned 3D latent, then decode into a set of Gaussians per occupied voxel. Upsampling keeps sparsity with a binary occupancy head; training uses ground-truth occupancy plus injected false positives. A light KL term regularizes the latent. The main setting uses latent dimension 32 and 63 Gaussians per occupied voxel.
The generator is a masked autoregressive transformer that jointly predicts occupancy and latent tokens. Occupancy uses a focal loss. Latent values use a token-wise diffusion head rather than a vector-quantized codebook. Training encodes a dense view set into an incomplete target latent, randomly masks some observed positions, and always treats unknown regions as prediction targets that are excluded from the loss. Inference grows a region: observed voxels seed a k-NN graph, BFS assigns depth, and a token is written only if occupancy clears a threshold. A coarse-to-fine occupancy stack scans the volume at high recall, then predicts latents only on surviving voxels.
The headline protocol is novel-view synthesis from two wide-baseline views at 224×224. 3DFront has 10k+ synthetic indoor scenes. RealEstate10K uses MASt3R-SfM for poses and pointmaps.
| Method | 3DFront FID | PSNR | SSIM | RE10K FID | PSNR |
| PixelSplat | 165 | 9.07 | 0.18 | 155 | 13.73 |
| DepthSplat | 110 | 13.76 | 0.49 | 82 | 13.25 |
| LatentSplat | 180 | 13.92 | 0.33 | 53 | 16.09 |
| MVSplat360 | 111 | 9.72 | 0.35 | 66 | 15.40 |
| SPAR3S | 59 | 15.18 | 0.62 | 41 | 16.72 |
On 3DFront, against the strongest generative baseline LatentSplat, FID falls from 180 to 59 and SSIM rises from 0.33 to 0.62. Adding conditioning views from 1 to 12 improves metrics monotonically. A lower occupancy threshold is safer: missing a wall hurts more than adding a spurious blob. Ablations: drop the diffusion head and PSNR goes 15.18→13.90; drop 3D RoPE →13.74; random autoregressive order →14.81; no occupancy refinement →14.48. Decoding with ground-truth occupancy reaches 17.90 PSNR, so occupancy errors still cap quality.
Scene completion happens natively in 3D. No novel cameras need to be sampled first, and multi-view consistency is not left to a 2D diffusion prior. Supervision is multi-view images only, which fits indoor data without 3D scans. The numbers stop at 224×224 indoor scenes. This is a feasibility result for sparse-view scene completion, not a production novel-view engine.
The appendix is blunt: if occupancy misses voxels inside walls or objects, the scene grows visible holes, currently the worst visual failure. Voxel resolution and Gaussians per voxel also produce local blur. The authors treat both as scaling issues and mention 2D diffusion post-processing, which they do not run. Main experiments are 224×224. 3DFront uses ground-truth cameras and rendered depth. RealEstate10K paths are smooth with large overlap, easier than the wide-baseline indoor case. Encoder and generator are trained separately. The main text says about four days each on one A100; the appendix lists about a week for the encoder plus 1.6M autoregressive steps on a V100. Budget planning should follow the appendix.