Light Field Primitive for Novel View Synthesis
Liang Chen, Jiahui Ning, Xun Jiang, Xing Xu, Jimmy Ren, Fenglei Fan, Heng Tao Shen
cs.CV
2026-09-25
LFP represents scenes as differentiable two-plane primitives, hitting 28.02 dB at 101 FPS on Mip-NeRF 360; one renderer also covers AA, defocus, fisheye, and refraction.
NeRF stores a scene as a continuous 3D radiance field and integrates along camera rays. 3DGS stores it as optimizable 3D Gaussians and projects them onto the image. Both pin the scene in 3D space. A camera does not measure a quantity at a point. It measures radiance arriving along each ray. The renderer has to translate: NeRF turns a ray into point samples, 3DGS turns each Gaussian into a screen-space footprint a pixel can evaluate.
Under a pinhole, one pixel is one ray, and that translation can still learn. Leave the ideal model and every failure is a statement about rays. Coarser resolution widens the bundle a pixel gathers, which is aliasing. A finite aperture spreads the bundle, which is defocus. Those statements do not map onto space-anchored variables, so each one rebuilds the renderer. Mip-Splatting adds filters in world and screen space. Deblur-NeRF adds an auxiliary network.
Classical light field rendering already indexes radiance by rays. Anti-aliasing is prefiltering the ray table. Defocus is averaging the rays a synthetic aperture admits. The table has to be dense, storage is heavy, novel views sit inside a fixed volume, and the lookup cannot be optimized against posed images.
LFP moves differentiable point-based optimization into 4D ray space. A pair of parallel planes indexes a ray by the four coordinates of its two intersections. Rays that cross a third parallel plane at one point form a ray group. Inside the group the two plane coordinates are coupled by an affine map whose slope is disparity: epipolar geometry written in light-field coordinates.
Each primitive is tied to one group and stores an anchor ray, a disparity, an anisotropic covariance, an opacity, a base color, and three anisotropic spherical Gaussian lobes. The anchor and disparity name the group. The covariance grades how a query falls off as it leaves the group. Base color holds the coherent radiance. Highlights go to the lobes, evaluated after reflecting the query about the anchor direction, which acts as a shading normal.
A camera ray is not projected, marched, or intersected with a 3D blob. The renderer compares the query to the group on the primitive's own disparity plane, turns the 2D residual into a Gaussian response, and alpha-composites in disparity order. With several atlases, disparity is no longer comparable, so the sort key is camera-frame depth at the peak of the response. Covariance is dilated by a half-pixel width so a primitive cannot be narrower than the view can resolve.
One plane pair cannot index rays that run nearly parallel to it. Before training, k-means on training-camera directions fixes 16 atlases, each with its own plane pair. A primitive keeps the atlas it was assigned. Learning the normals does not help: moving a normal shifts every residual on that atlas together, and the gradient cancels. COLMAP sparse points are starting values only. The loss is L1 plus D-SSIM, plus an L2 budget on highlight amplitudes so lobes do not steal radiance that belongs on the base. Densification watches gradients on the anchor and the disparity.
Anti-aliasing and defocus add a second-moment term to the covariance the renderer already reads, then rescale opacity by a determinant ratio when needed. Fisheye changes only the pixel-to-ray map. Refraction adds nothing. Grouping asks that radiance cohere, not what produced it.
Standard NVS follows the NerfBaselines protocol on one H100, 30k iterations per scene.
| Method | Mip-NeRF 360 PSNR | Tanks and Temples PSNR | FPS (360/T&T) | Memory (360/T&T) |
| Mip-NeRF 360 | 27.68 | 23.43 | 0.12 / 0.27 | 34.4 / 34.4 MB |
| 3DGS | 27.45 | 23.84 | 166 / 229 | 789 / 418 MB |
| Mip-Splatting | 27.49 | 23.93 | 149 / 203 | 782 / 452 MB |
| LFP | 28.02 | 24.38 | 101 / 173 | 313 / 150 MB |
LFP leads both datasets on PSNR and SSIM, and sits second on LPIPS. Against 3DGS it is +0.57 dB on Mip-NeRF 360, at about two thirds the frame rate and under half the memory.
Multi-scale follows Mip-Splatting: train once, test at 1x, 2x, 4x, 8x. Zoom-in average PSNR is 27.47 versus 27.37 for Mip-Splatting and 23.25 for 3DGS. Zoom-out is 31.92 versus 31.97 and 24.84. LFP adds no trained parameters; inference edits a term it already carries.
Defocus uses the ten real scenes from Deblur-NeRF. 3DGS and Mip-Splatting, which do not model the aperture, leave the blur in place. LFP scores 23.33 dB, third behind DOF-GS at 23.48 and Deblur-NeRF at 23.40, and second on SSIM at 0.728. Each training image gets two scalars, aperture and focal distance. Dropping the aperture term at test time yields a sharp render; sweeping focal distance refocuses after the fact.
On six ScanNet++ fisheye scenes, LFP scores 28.52 dB, just above 3DGUT at 28.47 and Fisheye-GS at 28.14. 3DGS has to run on undistorted images and falls to 23.70. On four glass-and-liquid scenes, LFP scores 23.66, ahead of RT-Splatting at 23.37 and Eikonal Fields at 21.54.
Ablations are on the nine Mip-NeRF 360 scenes. One atlas scores 21.10 dB, a 6.92 dB drop. Four atlases recover 6.62 of that, sixteen reach 28.02, thirty-two add 0.01. Three ASG lobes sit at the quality-speed knee: 28.02 dB at 101 FPS. One lobe costs 0.15 dB and runs at 120.6 FPS; five lobes score 27.97 dB at 84.4 FPS. Dropping highlight regularization costs 0.17 dB, lobe amplitude grows 6.6x, and the model shrinks 12% because the lobes absorb error that would have driven densification.
The extra 0.6 dB on the standard boards is incremental. The structural claim is the one to follow. Aliasing, defocus with refocus, fisheye, and refraction each rebuild a 3D-anchored renderer. In LFP each is a small edit to the same ray-space one. If the workload is already NVS and also hits resolution changes, shallow depth of field, fisheye lenses, or glass, this is a cleaner path than hanging a specialist module off 3DGS. There is no explicit surface, and frame rate still trails 3DGS.
LFP records rays, not matter, so nothing in it is a surface, and tasks that need explicit geometry are out. Atlas normals are frozen from the training cameras, a primitive never switches atlas, and coverage follows the capture rather than the content. Ordering across atlases falls back on camera-frame depth.
The lead on transparent scenes is 23.66 versus 23.37; the paper says refraction is far from solved. Specialized defocus methods still sit slightly higher. The closed-form aperture runs a few percent darker than sampling the disc, and thin bright features can lose about a third of their brightness. Atlas normals cancel when they are allowed to move, so large extrapolations in view direction may fail first. Unposed capture and feed-forward reconstruction are untested.