GRF-Recon: Global Ray-Field Optimization for Long-Sequence Feed-forward Reconstruction
Enpeng Li, Yunzhou Zhang, Zhiyao Zhang, Dexuan Lyu, Chenyu Wang, Chiyuan Cui, Cheng Cheng
ECCV 2026 as a Spotlight pre
cs.CV
2026-09-17
GRF-Recon adds LoRA and a sparse ray-field on Depth Anything 3. Uncalibrated KITTI mean ATE is 7.18 m, vs DA3-Streaming 12.72 m and calibrated ORB-SLAM3 7.93 m.
Feed-forward 3D foundation models such as DUSt3R, MASt3R, VGGT, and Depth Anything 3 can spit out poses and dense point clouds from a handful of uncalibrated views. Short clips look good. Stretch the same models to monocular driving video (thousands of frames, kilometers of road) and two failures hit at once. Transformer attention grows quadratically, so a consumer GPU OOMs after a few dozen frames. Local errors accumulate as Sim(3) drift, and junctions no longer close.
Chunk-and-align keeps the job alive, but the glue between chunks is thin. VGGT-Long, Pi-Long, and DA3-Streaming all live on that path, and long-trajectory accuracy drops. Enpeng Li, Yunzhou Zhang, and colleagues at Northeastern University, China, compress the job to one constraint: add cross-frame geometry that actually bites, inside a 24 GB memory budget.
The backbone is Depth Anything 3. GRF-Recon is three stages.
Local geometry first. Vanilla DA3 over-smooths occlusion boundaries on sequential video. The decoder that does cross-view matching stays frozen. Re-normalized LoRA (rank 8) is inserted only on the DINOv2 encoder QKV projections. Naive monocular distillation drifts feature norms and mismatches the frozen decoder, so the updated weights are rescaled to the original Frobenius norm. Trainable parameters grow by less than 1%; total parameter growth stays under 3%. Supervision is two-branch: a depth branch on MoGe teacher pseudo-labels, a point branch on ground-truth clouds, with λ=0.4.
Long sequences are chunked, default 60 frames with 10 frames of overlap. Neighboring chunks yield a Sim(3) from the overlap: SVD for rotation, least squares for scale and translation. That cascade is the coarse global trajectory.
The piece that holds drift is a hybrid-weight sparse ray-field. DA3's ray head emits a 7-D ray per pixel (direction, origin, confidence) and the depth head emits metric depth. The 3D point is origin plus depth times direction. Pixels are sampled by a blend of depth confidence, ray confidence, and depth gradient, then non-maximum suppression keeps them spread out. Matching cost is 0.6 on the perpendicular distance from a point to the target ray, 0.2 on gradient similarity, 0.2 on patch photometric error. A match survives only if bidirectional reprojection stays under 5 pixels.
Loop detection uses DINOv2 global descriptors. A new keyframe is cut when similarity to the last one falls below 0.70 (about every 3 to 5 frames on KITTI). A historical keyframe above 0.85 similarity and more than 100 frames away is a loop candidate. The global objective is a factor graph: sparse ray error plus multi-frame relative-pose priors (λpose=10), Huber loss, solved with Levenberg-Marquardt.
One RTX 3090. Images resized to width 504. Trajectories and clouds are Sim(3)-aligned to LiDAR before scoring. KITTI averages drop the high-speed Seq. 01, matching the paper's table.
| Method | Setting | KITTI mean ATE RMSE |
| ORB-SLAM3 | calibrated | 7.93 m |
| VGGT-Long | uncalibrated | 19.78 m |
| Pi-Long | uncalibrated | 19.01 m |
| DA3-Streaming | uncalibrated | 12.72 m |
| GRF-Recon | uncalibrated | 7.18 m |
On Seq. 00 (3724 m, 4542 frames) GRF-Recon is 4.35 m against DA3-Streaming at 9.34 m and VGGT-Long at 11.16 m. VGGT and stock DA3 OOM on these sequences. MASt3R-SLAM tracking-lost on every run.
Waymo nine-sequence mean ATE: GRF-Recon 1.39 m, DA3-Streaming 1.76 m, VGGT-Long 2.03 m. Front-end depth on KITTI moves Rel from 5.14 to 4.50 and δ1 from 92.4 to 97.1. Four-dataset mean Rel is 6.95, still above the MoGe teacher at 5.62.
Ablations pin the causal pieces. KITTI 00 full system 4.35 m; drop LoRA and it is 6.80; drop ray matching and it is 7.35; disable the whole backend and concatenate camera poses, 12.35 m. Chunk 30 vs 60 and overlap 10 vs 20 barely move the number, so 60/10 is the default. Seq. 00 end-to-end is 11 min 45 s. Feed-forward is about 8 s per chunk; ray matching and chunk alignment hide behind that; full LM is about 10 s.
For anyone trying to reconstruct kilometers of uncalibrated monocular video, this is a path that finishes on 24 GB: do not ram VGGT through a long sequence, freeze the decoder, LoRA in high-frequency geometry, then let sparse rays pin chunks together. On KITTI the trajectory number meets calibrated ORB-SLAM3, which is rare for the feed-forward line. The clouds can seed 3D Gaussian Splatting; the paper reports convergence around 1,000 iterations and skips a separate COLMAP stage.
It is still an incremental system paper. The backbone is off-the-shelf DA3, and chunk alignment is not new. The contribution is how the ray-field enters the global factor graph, plus the norm-preserving LoRA trick that stops distillation from breaking matching.
The authors say it themselves: rigid scenes are fine, highly dynamic objects are not. Seq. 01 (highway) is 76.52 m ATE, only a shade better than DA3-Streaming at 83.64 m, and dropping it from the average is an admission that the case is unsolved. Waymo LiDAR never sees rooftops or tree canopies, so valid visual structure gets punished by Accuracy; the number and the screenshot disagree. Every metric is after global Sim(3) alignment, so absolute scale is still borrowed. There is no indoor, handheld, or night long-sequence test. UAV and 3DGS results are qualitative, with no PSNR table. No code link is given.