NaCR: Visual Localization via NeRF-aided Camera Ray Regression
Yesheng Zhang, Xiang Dai, Xu Zhao, Chongyang Zhang
cs.CV
2026-09-22
NaCR uses NeRF for patch-level view synthesis and photometric backprop on camera rays, cutting 7-Scenes median error from 3.5 cm to 2.2 cm at 0.03 s.
Visual localization estimates a 6DoF pose for a query image in a known scene. Scene Coordinate Regression predicts unbounded 3D points per patch. Background points have huge variance, so training usually needs depth priors to mask outliers. DIMM switched the representation to Camera Ray Regression: each patch predicts a bounded camera ray (direction plus moment), then rotation and translation are solved as two linear problems, without depth priors. That helps outdoors; indoor accuracy stays mediocre. NeRF does the inverse map, rendering a patch from a ray. Shanghai Jiao Tong University joins the two at the ray.
Three DIMM upgrades come first. The backbone becomes the DINOv2 pretrained in Depth Anything 3, keeping a patch decoder so Gradient Decorrelation Training still applies. Three reference images (cluster centroids of training poses) condition each query patch token, turning absolute ray regression into a relative problem. A confidence head down-weights uncertain rays in training and weights RANSAC at inference.
NeRF then does two jobs. A Nerfacto is trained per scene; poses and appearance embeddings are jittered; BRISQUE drops bad renders. Stage 1 runs GDT on a 2:1 mix of real and synthetic patches, with a ray-confidence loss and an adversarial loss that aligns real/fake features (only the last two DINO attention layers unfrozen). Stage 2 starts once rays are stable: predicted rays go into Nerfacto, the whole patch is sampled, and photometric L2 flows back into the regressor. Adding that loss from step zero collapses training in highly non-convex RGB space, which is why the split exists. Optional NaCR+NPR then refines the pose by 2D-3D matching against rendered RGB-D.
Training uses an A800. Images are 504×378 with 14×14 patches. Stage 1 is AdamW at 1e-4 for 40K steps; Stage 2 drops to 1e-6 for 20K more. Batch size 5120.
On 7-Scenes with SfM poses, median error moves from DIMM's 3.5 cm / 1.0° to NaCR's 2.2 cm / 0.4°. GRLoc, which augments with 3DGS views, sits at 2.8 cm / 0.8°. Indoor SCR remains tighter: ACE averages 1.1 cm / 0.3°. NaCR+NPR reaches 0.3 cm / 0.2°, under RAP at 0.6 and GRLoc at 0.5. On outdoor Wayspots, 10 cm/5° recall is 54.5% for NaCR versus 42.9% for DIMM and 52.2% for ACE; 0.5 m/5° is 66.3% versus ACE at 62.6%. NPR can hurt on repetitive texture (Statue, Winter Sign).
| Method | 7-Scenes median | Wayspots 10cm/5° |
| DIMM | 3.5 cm / 1.0° | 42.9% |
| ACE | 1.1 cm / 0.3° | 52.2% |
| NaCR | 2.2 cm / 0.4° | 54.5% |
| NaCR+NPR | 0.3 cm / 0.2° | 54.9% |
Ablation averaged over seeds: architecture alone 3.2 cm, plus synthetic views 2.4 cm, plus rendering loss 2.2 cm. A single stage that includes rendering loss from the start falls to 6.7 cm / 3.3° on 7-Scenes. Per-scene mapping grows from DIMM's 1.32 h to 3.63 h (0.42 h of that is NeRF). Query inference is 0.03 s versus DIMM's 0.02 s; the forward path does not run NeRF.
CRR and NeRF share the ray interface, so synthetic views are consumed patch by patch, denser than supervising one pose per image. The photometric loop adds an image-space signal on top of geometric ray loss. Inference stays a single forward pass, which fits repeated queries if hours of mapping are acceptable. Without NPR, indoor accuracy still trails ACE. The gain is mapping-time data and supervision, not a faster localizer.
Every scene needs a usable NeRF. On 6-Scenes, large extent and lighting produce artifacts that poison both augmentation and the rendering loss, and NaCR falls below DIMM. Repetitive texture can send NPR the wrong match. Mapping takes 3.63 h with 39.53 M parameters, versus ACE at 5 minutes and 2.10 M. The two-stage schedule (when to switch, μ=0.1) is empirical. Joint NeRF-CRR training is left for later.