Many Eyes, One World: Feed-Forward 3D Reconstruction from Mixed Cameras
Qiaoge Li, Yifan Zhan, Haijun Yang, Haiyang Liu, Yiyi Cai, Chenchi Luo
cs.CV
2026-09-29
MEOW reconstructs mixed perspective, fisheye and panorama tuples in one pass from pixels only, scoring 80.4 mAA@30 on 2D3DS versus 54.3 for camera-labeled Wid3R.
Surveillance, consumer panoramas and robot heads often mix perspective, fisheye and 360° cameras in one reconstruction job. Feed-forward 3D models such as DUSt3R, VGGT, π³ and MapAnything were trained for perspective images. Their data mix and input pipeline do not cover heterogeneous projections or full panoramas. Rectifying a wide-angle frame to a pinhole view needs calibration and throws away field of view.
Wid3R jointly processes multiple camera models but needs a camera-class token on every view. CAM3R drops that requirement, then reconstructs one pair at a time and globally aligns. No published single-pass method takes a mixed tuple that includes full panoramas and returns a shared metric scene from pixels alone.
Reliable calibration is often missing in the field, and camera-type labels do not travel with the images.
MEOW treats mixed-camera reconstruction as data adaptation. The network stays the public MapAnything backbone; all later training uses synthetic mixed tuples.
LENSCOPE is the data engine. A procedural indoor generator builds 2,004 first-generation rooms and 499 denser second-generation scenes. Each optical centre is rendered once as an equirectangular panorama with analytic rays, metric depth and pose, then resampled through a continuous camera manifold. Seven models are drawn by weight: OpenCV with distortion 0.40, pinhole 0.15, Fisheye624 0.13, EUCM 0.07, Mei 0.05, spherical crop 0.05, full panorama 0.15. Field of view and pose follow photographic priors. An optics layer (blur, vignetting, noise) is decoupled from geometry so sharpness cannot leak the field of view.
After cameras are sampled, covisibility is recomputed: the fraction of 3D points mutually visible between two views. Only 2 to 8-view tuples whose graph is connected at edge weight 0.25 are kept. No real image of any camera type enters training after the public perspective pretraining.
Every image is resized anisotropically to one tensor shape with no crop, so a panorama keeps every longitude. Resize changes aspect ratio, so a 0.08%-parameter MLP adds log(width/height) to that view's patch tokens. Full panoramas are periodic in longitude; the dense head uses circular padding. At test time a pixel detector sets the panorama flag. It misses no panorama on the two headline sets and flags 2 of 386 other views.
The loss weights ray, depth and point terms by solid angle so panorama poles and fisheye rims do not dominate. Ray directions use a von Mises-Fisher likelihood on the sphere. Training has three stages: 35 epochs on first-generation renders with the official pipeline, 100 epochs with full-field resizing and online camera sampling, then two second-generation runs of 15 and 859 epochs interpolated at 0.25/0.75. One forward pass returns rays, depth, camera-to-world poses and a single metric scale. No alignment, matching or optimisation follows.
Heterogeneous Stanford 2D3DS: 88 tuples of 3 to 24 views cycling through full panoramas, 90° perspective frames and 180° equidistant fisheyes, one forward pass. mAA@30 is the mean, over thresholds from 1° to 30°, of the fraction of pairs whose relative rotation and translation-direction errors both pass.
| Method | Camera input | mAA@30 | RTA@30 | ATE↓ |
| MEOW | none | 80.4 | 96.2 | 0.62 |
| Wid3R | class per view | 54.3 | 86.5 | 0.83 |
| π³ | none | 19.8 | 59.7 | 1.24 |
| MapAnything | none | 16.4 | 53.9 | 1.48 |
| VGGT | none | 10.5 | 47.3 | 1.65 |
The gap is mostly translation. Rotation is already solved on both sides: Wid3R RRA@30 96.9, MEOW 95.2. Running public MapAnything with the same full-field resize moves the score from 16.4 to 15.4, so input resizing alone does not explain the gain.
A laser-scanned multi-room office (12 Leica BLK360 G2 stations) evaluates every method zero-shot. On the four-view mixed track MEOW registers every tuple, RRA@30 and RTA@30 both 100, AUC@30 79.4. Wid3R solves rotation (97.2) and fails translation (RTA@30 58.0, AUC@30 29.3). DUSt3R, MASt3R, π³, MapAnything and VGGT sit between 19.0 and 32.9 AUC@30. On pinhole-only tuples MEOW scores 85.7 against the MapAnything backbone at 86.8. On panorama-only tuples it scores 71.0; perspective models top out at 2.7.
Swapping in the public crop loader costs 3.3 mAA on 2D3DS and 5.0 AUC on the laser mixed track. Dropping the aspect-ratio embedding costs 23.2 and 11.3; 16:9 squeezed panoramas fall from 78.6 to 42.4. Eight views take 0.66 s on one RTX 5090, matching MapAnything; 32 views fit in one forward pass.
Split by tuple size, the picture is less even. MEOW leads on 3 to 8 views and loses on the four tuples of 15 to 24 views, 29.3 against Wid3R's 52.1. Pooled over all 4,044 ordered pairs those four tuples hold 49% of the pairs, and mAA@30 becomes 54.2 against 54.7. MEOW still wins 79 of 88 tuples.
For anyone mixing a phone, a fisheye and a panorama head on site, calibration, camera-type labels and a pairwise-then-align loop are no longer required: one pass returns metric pointmaps and poses. A perspective-pretrained backbone can be adapted to heterogeneous cameras with a procedural camera manifold and a covisibility check.
The working envelope is indoor, room scale, about eight views at a time. The data engine, laser benchmark and evaluation pipeline are planned for release.
This is incremental. Existing feed-forward models fail on mixed tuples: 10.5 to 19.8 mAA@30 for VGGT, π³ and MapAnything against 80.4 here.
The authors list the hard constraints. The engine covers synthetic indoor scenes, and the final model is adapted on room-scale tuples of at most eight views. On 2D3DS tuples of 15 to 24 views it already trails Wid3R. On cm-baseline perspective video it trails perspective models: 69.0 AUC@30 on Replica against 98.5 for π³ and 93.5 for VGGT. Scale-invariant metrics do not test predicted metric scale, which runs 12% to 25% short of truth (median ratio 1.14 on 2D3DS, 1.26 to 1.34 on the four laser tracks).
The evaluation is cleaner than wild mixed capture. Perspective and fisheye views are resampled from real panoramas with exact rays and metric ground truth. The laser set is one office, twelve stations, 24 four-view tuples per track: zero-shot holds, scene diversity does not. Failures cluster at low covisibility. On the laser panorama track the eight tuples that miss a pair at 30° have median minimum pairwise covisibility 0.042; the other sixteen sit at 0.424. Training required connectivity at 0.25, so those misses look out of distribution.
Specialist panorama models remain stronger on panorama-only input. PanoVGGT scores 99.9 AUC on 2D3DS panorama tuples (in-domain, trained on 2D3DS and Matterport3D), Wid3R 92.2, MEOW 75.4. Mixed tuples are the claim. Panorama-only is not.