TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations
Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta, Fan Zhang, Tanner Schmidt, Jakob Engel, Katerina Fragkiadaki, Adam W. Harley
cs.CV, cs.AI, cs.RO
2026-09-25
TrackEverything stores de-duplicated 3D tracks, beating open dense trackers by over 20 APD on TAPVid-3D clips and tracking every point past 1000 frames inside 40 GB.
Point tracking has been split in two. Sparse trackers last a long time but only for a handful of user queries. Dense trackers fill the frame and then run out of memory in a few dozen frames. The cause is per-frame 2D tokens: every reappearance of the same surface is encoded again. A 512×512, 200-frame clip is on the order of 10^10 predictions. Prior all-frame dense 3D trackers stop at 48–64 frames, then OOM.
Video is a 2D projection of a 3D world. Cost should grow with newly seen geometry, not with frame count.
TrackEverything runs 16-frame sliding windows. Each frame is encoded by frozen DINOv3-small plus a trainable ViT Adapter into a 1/4-resolution, 384-d feature map, then unprojected into a world-coordinate feature cloud with camera pose and depth from sensors or a feedforward reconstructor such as VGGT-Ω. Tokens are voxelized per frame inside the window so distinct surfaces that occupy the same place at different times are not merged too early.
Tracking is split. An endpoint refiner iteratively predicts each active point's 3D destination at the window boundary, plus visibility and static/dynamic class. Features come from 3D WAFT: bilinear samples of the 2D feature map at the birth location and at the current destination estimate, used as a template-matching signal instead of a 4D correlation volume. A trajectory refiner then decodes dense in-window paths only for dynamic points; static points stay put in world coordinates. At the window edge, tracks that land in the same voxel are mean-pooled and handed to the next window, together with newly revealed surfaces.
The model has 41M trainable parameters and 61M including the frozen 20M backbone. Training data are Kubric, PointOdyssey, and Dynamic Replica. Optional (t,x,y) query points are tracked without voxel merging.
On TAPVid-3D minival, mean APD-P over the first 48 frames is 34.7. Open all-frame dense VDPM scores 10.9, a gap above 20 points. Closed-source D4RT is higher on short DriveTrack and PStudio clips, with about 20× the parameters, private data, no public code, and no tracking past 48 frames. On full-length videos TrackEverything's mean APD-P is 31.2, against TAPIP-3D 29.8, CoTracker3 29.7, and DeltaV2 29.9, while tracking orders of magnitude more points. First-frame dense DeltaV2 never sees surfaces that appear later.
On PointOdyssey scaling plots, prior dense methods OOM around 96–200 frames; sparse methods track a few thousand points and OOM near 600 frames. TrackEverything stays under 30 GB at 900 frames. The paper presents it as the first 3D tracker that follows all visible points across 1000-plus-frame videos inside 40 GB.
Ablations: removing static/dynamic routing leaves APD-P at 31.2 but costs 4.8× latency and 2.5× peak memory (classifier F1 93.1 / 92.5 on Dynamic Replica / PointOdyssey). Dropping 3D WAFT falls to 29.3; dropping iterative refinement falls to 26.4. Voxel size 0.02 is the default trade-off; no voxelization OOMs. Switching geometry from Pi3 to VGGT-Ω lifts PStudio APD-P from 23.5 to 30.1; sensor geometry reaches 71.7.
For robotics, dynamic reconstruction, and video generation that need long-horizon, all-surface 3D correspondence, all-point long video becomes a single-GPU setting. The representation grows with scene content, not frame count. The geometry backend is swappable, so better sensors or reconstructors transfer without retraining the tracker.
Short-clip accuracy still tracks input pointmap quality. This is not a joint geometry-and-motion model. Against D4RT's short-clip numbers, capacity and data are smaller, and short clips are not a clean win.
Active memory still grows under perpetual open-world exploration; hour-scale trajectories need spatial eviction. External geometry is required, so pointmap errors become tracking errors. Boundary merges are irreversible mean-pools on spatial proximity, so a tracking mistake can weld distinct surfaces together. The query-point branch can protect landmarks; it is not a general fix. Training uses synthetic tracks, and harder real dynamics are still under-fitted.