OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields
Namitha Guruprasad, Abhay Yadav, Cheng Peng, Rama Chellappa, Anand Bhattad
cs.CV
2026-10-23
One image, no intrinsics: OmniRay's composite rays hit 0.14° on seen cameras and 1.12° on a withheld catadioptric lens, versus 6.34° and 75.7° for fine-tuned UniK3D.
Monocular 3D reconstruction assigns a 3D point to every pixel. Recent feed-forward models assume a pinhole camera, and the geometry bends on fisheye and panoramic images. Past 90 degrees from the optical axis a pinhole cannot represent the ray. Calibrating and undistorting first either crops the field of view or stretches the periphery, and depth models then report low confidence there.
Once a model predicts a ray per pixel, the parameterization is the bottleneck. UniK3D uses one spherical-harmonic basis for both the overall projection and the smaller deviations. For a roughly radially symmetric lens, one curve from image radius to incidence angle, rising monotonically, is enough. In a 2:1 equirectangular panorama a pixel closer to the image center can have a larger incidence angle. No monotone radial curve can reproduce that ordering.
OmniRay handles central cameras only: every viewing ray passes through one optical center. Each pixel becomes a unit ray d and a radial distance ρ, and the camera-frame point is P = ρ d. The predicted distance is measured along the ray. Perpendicular depth has a denominator that goes to 0 near 90 degrees and changes sign beyond that. Inference takes the image alone, with no intrinsics and no camera-type label.
A DINOv2 ViT-L/14 encoder feeds both branches. The ray module has three stages. Each residual rotates the previous ray on the sphere, and the residual heads start at zero.
The distance head is a DPT decoder over encoder blocks 4, 11, 17, and 23, with a softplus so ρ stays positive. The ray loss supervises the final field and both intermediate fields, with extra weight on the periphery and on the worst 5% of pixels. The point-cloud loss divides by each cloud's mean point norm, removing the global scale one image cannot fix. A wrong ray still moves the point.
Training turns the three ray stages on in order, then attaches the distance head and trains end to end. Images come from the 2D3DS and 360Loc panoramas, rendered as perspective, fisheye, and panoramic views. Random monotone radial profiles add more synthetic lenses, filled and annular. Samples near a held-out geometry are rejected. A GoPro fisheye and the Nayar catadioptric camera, a paraboloid mirror with orthographic projection covering an annulus of about 40 to 155 degrees, never enter training.
Before training, each representation is fitted to analytic ground-truth rays. The radial base reaches 0.002° on a fisheye and stalls at 22.8° on an equirectangular panorama. Raising the knot count from 16 to 32 barely moves that ceiling. Spherical harmonics at L=7 bring the panorama to 0.641° but sit at 0.276° on the fisheye, about two orders of magnitude behind the radial base. Radial base plus harmonic residual lands at 0.075°, 0.217°, and 0.174° on fisheye, panorama, and catadioptric. Among compact representations, only that composition stays under 0.25° on all three.
The benchmark compares GeoCalib, AnyCalib, Depth Anything 3, and UniK3D fine-tuned on the same 2D3DS and 360Loc panoramas. Ray error is the mean angle. Chamfer distance is the median nearest-neighbor distance after dividing by the scene's mean point norm. Every method gets per-image median scale alignment.
| Setting | OmniRay ray / CD | UniK3D | DA3 |
| Seen (pinhole/fisheye/panorama) | 0.140° / 0.0179 | 6.336° / 0.0760 | 33.028° / 0.1999 |
| Held-out catadioptric | 1.120° / 0.0131 | 75.702° / 0.5948 | 121.566° / 1.0412 |
| Held-out GoPro | 5.533° / 0.0392 | 6.342° / 0.0454 | 16.637° / 0.1097 |
| Unseen PanoCity | 1.350° / 0.0240 | 21.220° / 0.2120 | 50.533° / 0.3677 |
| Unseen CO3Dv2 | 2.967° / 0.0380 | 2.676° / 0.0277 | 2.041° / 0.0288 |
On the 4500-image pool, ray error is 1.414° against 20.211° for UniK3D and 47.457° for DA3. Relative CD is 0.0212 against 0.1736 and 0.3501. The 10 cm point-cloud F-score is 0.499 against 0.193 for UniK3D.
Without random-lens augmentation, seen cameras stay at 1.385°, while catadioptric and GoPro rise to 30.867° and 7.965°. Injecting rays into every DPT scale does not move metrics in one direction, so the final model drops it. With only the ray module trained, the staged three-part field reaches 0.399°, 1.322°, and 1.538° on panorama, fisheye, and catadioptric. A VGGT initialization records the best pinhole error, 0.330°, and is worse on the other four geometries.
CO3Dv2 is object-centric, and its COLMAP ground truth covers about 12% of pixels. OmniRay loses there: AbsRel 0.0889 versus 0.0573 for DA3 and 0.0478 for UniK3D, and F-score 0.173 versus 0.282 and 0.293.
Depth Any Camera and UniDAC warp each image into an equirectangular view before estimating depth, and that warp needs intrinsics. OmniRay, like UniK3D, takes one image and returns an up-to-scale point cloud in the camera frame. Against fine-tuned UniK3D, the held-out catadioptric error falls from 75.7° to 1.12°, and PanoCity from 21.2° to 1.35°.
On object-centric pinhole images it trails DA3. On GoPro, ray error only moves from 6.34° to 5.53°, and the 10 cm F-score falls from 0.235 to 0.215. It is the right tool when the lens is unknown and possibly odd. A calibrated pinhole pipeline does not need the switch. There is no metric scale, so the cloud is not a rangefinder.
Training views sit near the ground, with horizons roughly aligned. On an aerial panorama whose horizon is near the top of the frame, predicted rays cover a bounded forward region and miss the rest of the sphere. Altitude and camera orientation in the training set bound generalization as well. Multi-view reconstruction is left for later.
Every reported number uses per-image median scale alignment, so the comparison is about shape. The gap versus DA3, GeoCalib, and AnyCalib also mixes in the training distribution: UniK3D was fine-tuned on the same panoramas, and the other three were not. Held-out cameras and PanoCity are the fairer contrasts.
The GoPro filter only rejects profiles within 15° of its incidence curve at every radius. Distance from that lens to the training distribution is not otherwise measured. Non-central cameras sit outside the assumption. Oracle errors are fitting ceilings, and network error is more than an order of magnitude higher, so capacity is not used up. How much of the CO3Dv2 regression comes from ground truth on 12% of pixels is not separated.