Marrying Optimal Transport and ODEs for Unified Continuous-Time 4D Reconstruction and Tracking
Liying Yang, Hao Mo, Jialun Liu, Chen Liu, Xinxing Yu, Chenhao Guan, Hui Ma, Xiao Cao, Ajian Liu, Yanyan Liang
cs.CV
2026-08-10
Uni4R uses optimal transport and ODEs to learn a continuous velocity field for 4D reconstruction and tracking at any timestamp. SOTA on three benchmarks, 80 frames in 1.57 s.
4D reconstruction (a sequence of per-frame 3D pointmaps) and point tracking (the trajectory of the same points over time) are usually tackled together. But almost all current methods work only at integer frames: you feed frames 0, 1, 2 and get points and tracks at those frames. What about frame 0.7? The dominant answer is after-the-fact interpolation. TraceAnything, for instance, fits a B-spline through control points to fill the gaps.
The trouble is that a B-spline is pure curve fitting with no notion of motion. It does not know that points should travel along energy-minimizing, geometrically continuous paths, and at fractional timestamps it routinely distorts structure: wheels go elliptical, windows stretch. Worse, treating motion as mere fitting throws away a two-way relationship that should exist by right, where reconstruction constrains tracking with geometry and tracking guides reconstruction with motion.
Uni4R puts both tasks inside one continuous kinematic framework, so the model returns kinematically consistent results at any timestamp.
The core is to learn a continuous velocity field. Uni4R writes 4D dynamics as a sequence of local ODE initial value problems. The position of any point between frame k and k plus a fraction is the anchor P(k) plus an integral of the velocity field: P(k plus tau) = P(k) + integral of V(P(t), t) dt.
Using per-frame anchors instead of integrating from frame 0 all the way through keeps errors from accumulating. Each integer frame is a fresh anchor, so integration error does not compound across the sequence.
The hard part is that recovering a continuous velocity field from observations at integer frames alone is a badly ill-posed inverse problem. Infinitely many curved velocity fields can move a point correctly between frame k and k+1, so the model overfits and trajectories go unstable. Uni4R's fix is a kinematic inductive bias that narrows the solution space to the most kinematically consistent path.
The bias comes from Optimal Transport. Borrowing Flow Matching theory, the probability path that corresponds to optimal transport between two states is a straight line, which minimizes kinetic energy. Uni4R constructs this OT straight line on the anchor feature manifold as a prior on the global velocity direction. Inside the Flow Matching Guided Decoder (FMGD), a global velocity branch extracts anchor features X, a point reconstruction branch extracts geometric features FR, and an OT-path convex combination builds the FM-guided velocity features. A local velocity prediction module (LVP) fuses these with continuous time embeddings through spatial-temporal cross-attention and decodes the velocity at any timestamp. A first-order Euler ODE solver then integrates to get point positions.
Training hits another wall: there is no ground-truth velocity at fractional frames. The integral-consistency training strategy integrates predicted velocities from integer frame T to T+1 and supervises against the ground-truth pointmap at T+1. Gradients backpropagate through the integration path into the upstream decoder, turning an unobservable velocity field into explicitly supervised pointmaps.
Uni4R takes SOTA on all three axes.
| Task | Metric | Uni4R | Best competitor |
| World-coordinate 3D tracking (Point Odyssey) | APD3D up | 81.59 | V-DPM 81.27 |
| World-coordinate 3D tracking (Aerial Digital Twin) | APD3D up | 86.55 | V-DPM 85.72 |
| 4D reconstruction (Point Odyssey) | APD up / EPE down | 80.71 / 0.222 | St4RTrack 78.73 / 0.241 |
| Continuous-time kinematic coherence | MAM down | 7.272 | TraceAnything 7.891 |
On point tracking, Uni4R posts the highest APD3D across all four test sets (PO/DR/ADT/PStudio), edging V-DPM and beating St4RTrack and TraceAnything by well over ten points. 4D reconstruction is best across the board. The most telling result is on the new continuous-time kinematic benchmark (100 dynamic scenes, each with 24 integer and 230 fractional frames and dense ground truth): Uni4R is best on temporal smoothness (MAM), spatial topology stability (ELV, VLPD), and accuracy (APD), confirming that trajectories hold their geometric structure at fractional time instead of warping like the baselines.
On efficiency, 80 input frames run in 1.57 s on a single H800, 66.52% faster than TraceAnything, second only to St4RTrack (1.34 s) but far more accurate, while V-DPM takes 194 s.
The ablations tell a clean story: removing the flow matching loss, the FM velocity features, the integral-consistency training, or the LVP module each degrades performance; swapping anchor-based ODE for global ODE (integrating from frame 0) lets error accumulate and drops every task. Training only the velocity branch or only the reconstruction branch is worse than joint training, confirming that reconstruction and tracking constrain each other.
For anyone doing dynamic scene understanding (robotics, AR/VR, video effects, motion estimation in autonomous driving), this turns "what should the fractional frame look like" from a hand-wavy interpolation into a learnable problem with a kinematic constraint. When you need dense correspondences at arbitrary timestamps, you no longer have to swallow B-spline's geometric distortions.
The broader transfer is methodological: regularizing a badly ill-posed continuous inverse problem with the optimal-transport prior that a straight path means minimum kinetic energy. That idea is not specific to 4D; any task that recovers a continuous field from discrete observations could borrow it.
The paper has no dedicated limitations section, which is itself a weakness. A few points worth pressing: the model takes only monocular input, so the depth prior is inherently limited; training uses just three datasets (Point Odyssey, Dynamic Replica, Kubric), the first two largely synthetic, so generalization to messy real scenes is underverified; and the continuous-time benchmark is the authors' own, with metrics and scenes they defined, so there is no independent third-party yardstick. The 43.67 GB VRAM footprint is higher than TraceAnything (35.71 GB) and St4RTrack (24 GB), so deployment is not cheap. And while Uni4R is two orders of magnitude faster than V-DPM, V-DPM's accuracy is close, so the real engineering call depends on the precision-speed tradeoff.