VoxelTTO: Voxel-Aligned Feed-Forward 3D Gaussian Splatting with Test-Time Optimization
Yibin Zhao, Yihan Pan, Yangwen Li, Jun Nan, Jianjun Yi
cs.CV
2026-09-18
VoxelTTO pools VFM features into voxels, decodes Gaussians, and adapts LoRA at test time from pose. Replica test-view PSNR is 28.54 versus 22.82 for DepthAnything3, after 80 GPU-hours of training.
Feed-forward 3D Gaussian Splatting often grows one Gaussian per pixel. Pixel alignment produces overlap and artifacts, and a slightly wrong camera pose misaligns novel-view synthesis. Visual foundation models can emit depth, point maps, and pose from an arbitrary image set, but fine geometry and appearance stay soft. Some methods inject ground-truth cameras into the backbone, which changes the architecture, and the decoded pose still disagrees with the prior.
VoxelTTO targets geometrically cleaner 3DGS from any number of images, with camera parameters optional, and tighter test-view alignment when those parameters are present.
The backbone follows a DA3-GIANT recipe: DINOv2 patches, 40 layers of alternating attention, frozen. A feature DPT upsamples tokens to dense pixels, which are back-projected with predicted depth and cameras into a colored, featured, confidence-weighted point cloud. Points pool into voxels whose centers are confidence-weighted averages, not forced onto the grid; voxel confidence is the minimum inside the cell so weak points drop out. A sparse-convolution U-Net updates voxel features. Separate appearance and geometry MLPs decode n Gaussians per voxel, default n=2, so shape and color are not one entangled regression.
Rendering replaces vanilla 3DGS rasterization with stochastic solid volume rendering. With a suitable vacancy function, color nearly matches 3DGS; depth is the median where transmittance hits 0.5, and backpropagation differentiates that implicit condition. Figure 3 shows similar RGB and fewer holes or blurry depth edges.
Test-time optimization runs only when aligned cameras are given. LoRA sits on Q, K, V of layers 13–40, with a shared down-projection A and per-layer B. The loss covers relative rotation, scale-normalized translation, and intrinsics after pinning the first camera to identity. Default 20 steps. Training updates only the voxel-aligned reconstruction head: 75.3M trainable parameters out of 1.39B. Thirty thousand steps on ScanNet, Infinigen, ARKitScenes, and DL3DV-10K take 40 hours on two RTX PRO 6000s, the 80 GPU-hours in the abstract.
Replica, Tanks and Temples, and DTU, against AnySplat, YoNoSplat, DepthAnything3, MapAnything, plus camera-using VolSplat and MVSplat.
| Setting | VoxelTTO | Strongest baseline |
| Replica test-view PSNR | 28.54 | DepthAnything3 22.82 |
| Replica test depth RMSE (cm) | 6.945 | MapAnything 37.96 |
| Replica mean rotation error | 0.220° | DepthAnything3 0.324° |
| DTU overall Chamfer | 3.416 | 7.357 without TTO |
Without TTO, Replica test PSNR is already 25.83; TTO lifts it to 28.54. On Tanks and Temples, TTO moves test PSNR from 14.71 to 19.23 while input-view PSNR falls from 23.74 to 20.89, so aligning test views costs some input-view appearance. Replica depth is strong: 6.167 cm RMSE on inputs, 6.945 cm on tests. On TAT, input depth gets slightly worse with TTO while test depth improves.
Ablations: two Gaussians per voxel beat one; four barely help. Volume rendering on both train and test yields 28.54 PSNR; switching both stages to 3DGS rasterization drops to 21.52. Voxel size 0.002 m is the compute/quality compromise; 0.005 m falls to 23.44 PSNR. Ten 518² images without TTO take about 2.30 s, matching DA3; 20 TTO steps keep the total under 7.4 s.
“Pool to voxels, then split Gaussians” is a clean way out of pixel-aligned overlap. TTO does not rewrite the VFM or inject camera tokens, so an existing VGGT-style model can attach to posed scenes. Eighty GPU-hours and 75.3M trainable weights is a followable reproduction budget.
This is not optimization-free magic. TTO needs cameras; without them the method inherits the frozen VFM pose. The TAT input-view drop shows test-time alignment moves the representation rather than adding a free lunch.
There is no standalone limitations section; the tables already leak several. Input-view PSNR on TAT falls after TTO, even as the paper headlines an overall gain, so both columns need to be read together. Training samples sequential video, and there is no dedicated stress test on unordered photo collections. The 0.002 m voxel size is scene-scale specific. Stochastic volume rendering also replaces 3DGS at inference, so the output is not a drop-in Gaussian player. Code is on anonymous repos. Camera priors come from ground truth or external SfM; TTO does not estimate pose from scratch.