RLSplat: Resource-Linear Budget Control for Feed-forward 3D Gaussian Splatting
Injae Kim, Minseok Joo, Jihwan Park, Sehyung Kim, Minseong Bae, Hyunwoo J. Kim
cs.CV, cs.AI
2026-10-20
RLSplat decodes B Gaussians, not a pruned dense set. On zero-shot ACID it beats F4 Splat at every budget, still leads with 4× fewer, and at 2K is 4.1× faster with 9.6× less VRAM.
Feed-forward 3D Gaussian Splatting (FF3DGS) predicts a scene in one pass and skips the minutes-to-hours of per-scene optimization. The primitive count does not flex with the device. Pixel-aligned models drop a fixed set on every pixel, so overlapping views pile up redundant Gaussians. Query models such as C3G and GlobalSplat bake the count into the architecture. A new budget means a new training run.
EcoSplat predicts one Gaussian per pixel and keeps the B with the highest opacity. F4 Splat predicts several scales and selects B. The file gets smaller. The forward pass does not. Both build a candidate set much larger than B and only then prune or select, so latency and peak memory stay almost flat as the budget falls.
RLSplat, from KAIST and Korea University, ties the Gaussian count, inference latency, and peak memory to one budget B.
Uncalibrated context images and a budget B go in. A geometry backbone estimates the cameras and packs the images into anchor tokens, one per local 3D region. Every Gaussian is spawned from an anchor.
The policy predicts a proportion per anchor. In training those proportions drive a multinomial draw of B trials, so the integer counts sum to B. The draw is not differentiable, and quality appears only after rendering, so the update is GSPO: group-relative advantages, with the importance ratio taken over the whole allocation and clipped. No value network. The reward is the negative novel-view loss, pixel MSE plus LPIPS. At inference the expected counts are rounded to integers that still sum to B.
The decoder never builds a denser set. It predicts an initial Gaussian, then refines it with local cross-attention. Global self-attention would be quadratic in B. Each Gaussian attends to 16 neighbors, so cost tracks B. Neighbors come from a two-level search: 16 nearby anchors by centroid, then the 16 closest Gaussians inside those anchors. The search uses the initial centers, runs once, and is shared across layers.
If a Gaussian drifts out of its parent anchor's neighborhood, that search returns the wrong neighbors. A Voronoi loss pulls each initial center toward its own anchor centroid. An offset loss caps the residual shift at half the local anchor spacing. A projection loss ties Gaussians back to the source image patch and penalizes points behind the camera. Without the projection loss, training collapses.
Training has two stages. For 40,000 steps the decoder sees random allocations, with budgets from 2048 up to 16384 times the view count. The next 5,000 steps add the policy, starting from a uniform distribution and sampling low budgets more often. The base stage has no reference policy, so there is no KL term.
The model is trained on RealEstate10K and tested zero-shot on 249 aerial ACID scenes at 256×256. Latency and peak VRAM cover Gaussian generation only, on one RTX A6000. Loading, pose alignment, and rendering are excluded. In the table, 1K means 1024, and the budget is per context view.
| Method (8 views) | PSNR↑ | SSIM↑ | LPIPS↓ |
| EcoSplat, 4K/view (32K) | 23.29 | 0.669 | 0.300 |
| F4 Splat, 4K/view (32K) | 24.50 | 0.718 | 0.240 |
| RLSplat, 4K/view (32K) | 25.24 | 0.756 | 0.211 |
| EcoSplat, 1K/view (8K) | 18.99 | 0.533 | 0.456 |
| F4 Splat, 1K/view (8K) | 23.61 | 0.664 | 0.307 |
| RLSplat, 1K/view (8K) | 24.82 | 0.734 | 0.254 |
On 8 views the PSNR gap over F4 Splat grows from 0.74 dB at 4K per view to 0.99 dB at 2K per view. At 1K per view the paper states 1.22 dB (24.82 vs 23.61 in the table). That 1K model already beats F4 Splat at 4K per view, 24.82 against 24.50, with a quarter of the Gaussians. The gap also widens as the budget tightens at 16 and 24 views.
Fixed-count feed-forward models spend far more primitives. With 24 views, RLSplat reaches 25.58 dB with 24K Gaussians. SPFSplat reaches 25.00 dB with 1536K, 64 times as many. With 8 views, NoPoSplat and SPFSplat use 512K (22.84 and 24.89 dB) and AnySplat uses 470K (23.30 dB). RLSplat uses 32K and scores 25.24 dB.
At the 2K point in Figure 1, PSNR is 0.9 dB above F4 Splat, peak VRAM is 9.6× lower, and generation runs at 4.1× the speed. EcoSplat and F4 Splat stay nearly flat in latency as B falls. At 65,536 Gaussians, hierarchical search takes 8.1 ms against 219.6 ms for an exact search, and end-to-end generation drops from 595.0 ms to 388.0 ms. With the Voronoi loss, 99.0% of Gaussians get the same neighbors as the exact search. Without it, 60.2%.
On a RealEstate10K stair with 8K Gaussians total, the thin structure receives 2,521 Gaussians from EcoSplat (local PSNR 17.79), 3,026 from F4 Splat (20.00), and 5,388 from RLSplat (23.51). Full-image PSNR is 21.82, 23.42, and 26.51.
Architecture ablations use a shorter schedule, still on ACID with 8 views and budget 8192. The full model scores 22.84 dB. Dropping the offset loss costs 0.53 dB. Dropping the Voronoi loss as well lands at 20.33 dB. Removing local refinement costs 0.96 dB. Removing the projection loss falls to 5.80 dB. Allocation, fine-tuned from the full base model, is a smaller gap: uniform 24.53 dB, a Sobel-edge heuristic 24.76 dB, the learned policy 24.83 dB. The average edge over that heuristic is 0.07 dB. On one scene the policy cuts a flat wall from 165 Gaussians to 79 and raises a textured floor from 84 to 116. Full-image PSNR moves from 23.36 to 25.32. The neighbor count can be changed at inference with no retraining.
On-device reconstruction runs out of memory and time in the forward pass, before file size matters. A budget that only shrinks the saved Gaussians leaves generation cost almost unchanged. RLSplat makes one integer control both the representation and the cost of producing it. At the low end it runs at about 4× the speed of the baselines, with nearly 10× less peak memory, and it leads every budget-controllable comparison in the paper.
Neighbor count is a second knob on a frozen checkpoint. AR, robotics, and 3D vision-language models that need a scene built on the spot can pick a budget per device without another training run.
The learned policy is a small slice of the gain. Against an edge heuristic it adds 0.07 dB on average. Most of the quality comes from emitting exactly B Gaussians and keeping them inside the neighborhood the search can see.
The headline numbers are a zero-shot jump from indoor video to aerial footage at 256×256. There is no matched table on the RealEstate10K test split, so the indoor margin is not pinned down the same way.
The 4.1× and 9.6× figures cover Gaussian generation only. Rendering is excluded on purpose. Fewer Gaussians should also rasterize faster, but latency to a displayed novel view is not reported. 4.1× is not a number a viewer can time.
The geometry backbone is off the shelf. Camera and anchor error sit outside the ablations. Residual motion is capped at half the local anchor spacing, so a bad initial center that leaves the neighborhood will not be repaired by local attention. ZipSplat, a concurrent clustering approach to the same budget problem, is not compared. Robotics and other downstream uses have no experiments. The training cost of the grouped rollouts versus EcoSplat and F4 Splat is not reported.
A Sobel heuristic already reaches 24.76 dB, against 24.83 dB for the policy. A deployment that only needs a non-uniform split may get most of the image quality from that cheaper rule. Its latency and memory are absent from the main comparison.