ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild
Ilya Fradlin, Christian Schmidt, Jens Piekenbrinck, Karim Knaebel, Gonzalo Martin Garcia, Bastian Leibe
www
cs.CV
2026-10-01
ARROW unifies 3D reconstruction and point tracking over arbitrary image sets, static photos to multi-camera streams, setting new state of the art on WorldTrack and TAPVid-3D.
3D reconstruction and 3D point tracking grew up as separate lines. Reconstruction, from classical SfM to feed-forward geometry transformers like DUSt3R, VGGT, and Depth Anything 3 (DA3), takes unposed image collections where order does not matter. Point trackers, from TAPIR to CoTracker, assume a temporally continuous monocular video. The two problems are deeply connected: geometry constrains correspondence, and correspondence carries structure across views and time. Yet each kept its own models.
D4RT merged geometry and tracking into one model: encode a video once, then answer queries specifying a source pixel, a target time, and a reference camera. Its input side stayed video-native, using learned timestep embeddings to tie each query to a frame. That breaks once inputs are not a video. Multi-camera streams share capture times, static collections have no frame order at all, and training only on temporally adjacent observations narrows the correspondence problems the model ever sees.
ARROW takes an unordered set of 2 to 24 RGB images, any viewpoints and capture times, and answers the same relational query as D4RT: given a pixel (u, v) in a source observation, return its 3D position at the target observation's capture time, expressed in the reference observation's coordinate system. Vary the three roles and the same interface yields tracks, point clouds, or camera poses.
The design problem is how a query finds its frames. D4RT imposes order with positional encodings and addresses frames by learned timestep embeddings; swap in an order-free geometry-transformer context and those fixed embeddings have nothing to match. ARROW's fix repurposes DA3's per-image camera tokens as ID tokens: embeddings derived from each frame's content, living in the same embedding space as the global context, so a query's ID tokens align with the right frames regardless of input order.
Two components serve this:
The training recipe matters as much as the architecture. Static multi-view sets give correspondence for free, dynamic data with tracks gives motion supervision, and trackless data like nuScenes is restricted to same-time queries for geometry-only supervision. Sampling is deliberately non-sequential: candidate windows stretch up to 10× the requested clip length, and multi-view datasets mix cameras. The mixture spans 19 datasets, about 78K scenes and 22.8M frames, initialized from DA3-Giant and trained for 200k steps on 32 GH200 GPUs, with D4RT's multi-task loss.
WorldTrack monocular 3D tracking (64 frames; APD = share of points within distance thresholds):
| Dataset | ARROW APD | Best prior APD |
| Aria Digital Twin | 93.55 | 91.75 (SpatialTrackerV2) |
| Dynamic Replica | 88.00 | 84.41 (4RC) |
| PointOdyssey | 85.26 | 83.84 (UniQuery4R) |
| Panoptic Studio | 85.76 | 84.76 (UniQuery4R) |
ARROW leads on APD and EPE, overall and dynamic-only, on all four datasets.
The multi-view gap is the widest. On an RGB-only adaptation of the MVTracker benchmark (four static cameras, 24 synchronized timesteps, 2D queries instead of supplied depth and calibration), ARROW averages 61.0 APD and 0.1252 EPE against 48.0 and 0.1701 for OmniX, the runner-up, leading 11 of 12 dataset-metric combinations. The sole exception is DexYCB EPE: 0.1705 versus 0.0991 for TAPIP3D + VGGT-Ω. On TAPVid-3D, ARROW posts the best average camera-coordinate and world-coordinate AJ and APD, trailing slightly on occlusion accuracy.
Cumulative ablations isolate the contributions: identity-based addressing improves tracking over a D4RT-style ordered baseline; removing encoder temporal embeddings leaves monocular tracking nearly unchanged; non-sequential sampling adds 28.85 percentage points of multi-view APD in a single step, the largest gain; four extra static multi-view datasets (15 to 19) lift every reported metric further. On reconstruction, ARROW takes the best NRGBD Chamfer (0.0263) and normal consistency (0.933), the lowest Sintel ATE (0.0714), and the lowest KITTI AbsRel (0.036), while trailing OmniX slightly on 7-Scenes. Against dedicated reconstruction models (VGGT, π3, DA3) it is on par, not ahead.
For anyone building 4D vision pipelines, this consolidates the interface: one open-source model ingests unordered photo collections, monocular video, and synchronized multi-camera streams, and outputs tracks, point clouds, poses, and depth without external depth or calibration. Multi-camera capture previously required dedicated stacks; here it is a native capability of a feed-forward pass.
The training finding is the more portable one: making the training distribution harder, with cross-view and long-gap sampling, improved plain temporal tracking enough to set the state of the art. Supervision diversity, as much as architecture, is where tracking models still have headroom.
Honest framing: reconstruction is a tie, and the tracking margins matter most for long sequences and multi-view inputs. If your workload is short monocular clips, the gains are modest.
From the authors: dropping explicit time encodings means ARROW only produces outputs at the capture times of its input observations, with no interpolation between frames and no extrapolation to the future. Dense decoding is more expensive than dedicated dense heads. Queries do not interact, which may weaken low-confidence and far-away regions. Mixing in-distribution views with out-of-distribution inputs such as egocentric cameras degrades multi-view performance, and fine-grained pose and (u, v) precision remains sensitive to local variations, with a few pixels of UV error on TAPVid-3D.
Two caveats from reading the paper. The RGB-only MVTracker protocol is the authors' own adaptation, and they state it cannot be compared numerically with official MVTracker results, so the claim of beating dedicated multi-view trackers rests on that adapted setup. The WorldTrack table lacks the original D4RT; the stand-in OpenD4RT is an unofficial reimplementation scoring 71.92 APD on ADT, a weak link in that comparison. Reproduction is not cheap either: 200k steps on 32 GH200 GPUs plus manual per-scene review of PointOdyssey annotations, though code and weights are public.