Info3R: Information-Adaptive Test-Time Training for 3D Reconstruction
Sunghyun Baek, Hanna Bae, Minchan Kwon, Junmo Kim
cs.CV
2026-09-18
KAIST adds training-free, information-gated test-time updates and a confidence-triggered state reset on CUT3R. Mean KITTI Odometry ATE is 30.48 versus LongStream 51.24.
CUT3R keeps a finite set of state tokens and, for each incoming frame, cross-attends with the image, updates the state, and emits point maps plus pose. TTT3R recasts that update as test-time training: a spatial-alignment confidence becomes a per-token learning rate, so a redundant frame no longer overwrites the whole state.
Two signals are still missing. Frame importance is barely differentiated, so motion blur, near-duplicates, and frames that actually add geometry are absorbed at similar strength. State capacity is finite; after a long stream it saturates, plasticity for new observations drops, and world-coordinate confidence collapses. LongStream fights ultra-long outdoor sequences with keyframe-relative poses and periodic refreshes, at the cost of extra training and much more memory.
Whether the state should update, and whether it should be thrown away, ought to depend on how much new information the current frame carries.
Info3R is a training-free test-time rule on top of CUT3R, in two parts.
The first part scales the learning rate by information. The redundancy weight is wredundant = 1 minus the mean cosine similarity of spatial tokens in consecutive frames: high overlap means a small update, so finite capacity is saved for novel views. Difference alone treats motion blur as novelty. So the method SVD-decomposes the current frame tokens, takes the effective rank reff that covers 95% of the energy, and sets winfo = min(reff / τ, 1) with τ fixed at 100. Blurry or collapsed frames have low rank and a suppressed update. The learning rate is αt = min(αbase · wredundant · winfo, 1), and the state interpolates as St = S{t-1} + αt (Ŝt − S{t-1}). TTT3R's βt comes from alignment confidence; αt comes from how new and how trustworthy the frame is.
The second part is a dynamic reset. A finite state still fills up even with selective updates. The trigger Γt sums αi / C̄worldi since the last reset: the numerator is information poured into the state, the denominator is world-frame confidence, so a confidence collapse inflates Γ faster. Past a threshold γ the state is re-initialized from the current image tokens, and the world head then writes in a new anchor frame Ac. Consecutive anchors are chained by the SE(3) the model predicts at the reset step, T{A0←Ac} = T{A0←A{c-1}} · T{A{c-1}←Ac}, and points plus poses are mapped back to the first-frame world. Resets cluster on turning segments and almost never fire on straight cruise, which is when the stored scene stops matching the new viewpoint.
Hyperparameters are per dataset: KITTI Odometry uses γ=3.2 and αbase=1.7; ScanNet γ=3, αbase=1.0; TUM γ=20; 7-Scenes γ=25. The backbone is CUT3R's ViT-Large on one RTX 4090.
Mean ATE on KITTI Odometry:
| Method | Mean ATE↓ | Seq.00 | Seq.01 | Seq.07 |
| CUT3R | 203.29 | 187.79 | 638.10 | 73.47 |
| TTT3R | 179.08 | 161.60 | 537.34 | 69.33 |
| LongStream | 51.24 | 77.05 | 49.45 | 15.66 |
| Info3R | 30.48 | 30.03 | 97.20 | 7.64 |
Mean ATE is 1.68× lower than LongStream (30.48 vs 51.24). Seq.00 returns to the start without an explicit loop-closure term. On short, simple ego-motion LongStream still wins: Seq.01 is 49.45 versus 97.20, and Seq.03/04 follow the same pattern.
Indoor pose at 1000 frames: ScanNet ATE falls from TTT3R's 0.394 to 0.176, about 54%; TUM from 0.117 to 0.061. At 50 frames the two methods are nearly tied. Length is what opens the gap.
Video depth on Bonn at 500 frames: Abs Rel 0.077 and δ<1.25 0.955 (TTT3R 0.100 / 0.922). KITTI at 500: 0.116 / 0.881 (TTT3R 0.132 / 0.866). 3D reconstruction on 7-Scenes at 400 frames: Acc 0.020 (TTT3R 0.049, CUT3R 0.163), almost flat in length; NRGBD Acc 0.072 versus TTT3R 0.141. Normal consistency is occasionally a hair below TTT3R.
On Seq.00, memory is 3.24 GB, matching CUT3R's 3.23; TTT3R uses 4.70, LongStream 10.77. FPS is 13.27, slower than CUT3R at 15.27 and LongStream at 20.47. That is the speed paid for the ATE.
Ablation: drop the redundancy term and ATE rises from 30.48 to 45.03; drop effective rank and it is 32.47. Drop cumulative α or confidence from the reset trigger and ATE lands at 36.68 and 36.18. Redundancy matters more for global trajectory: RPE can even look slightly better without it, ATE gets clearly worse.
If CUT3R or TTT3R is already in the loop, this is a no-retrain patch whose memory barely moves. On kilometre-scale outdoor sequences it undercuts the extra-trained LongStream on mean ATE at about 30% of the memory. The bill is a bit of FPS and a pair of dataset-tuned knobs, γ and αbase.
For online 3D, effective rank as a quality probe and cumulative update mass as a saturation probe fit the input more tightly than a fixed refresh interval. Resetting on turns and leaving the state alone on straights is the saturation detector doing its job.
γ ranges from 0.3 on KITTI depth to 25 on 7-Scenes; αbase from 0.4 to 1.7. There is no universal default. A new domain almost certainly needs a search.
Chained SE(3) alignment can accumulate error at reset points; the paper never reports the residual of that alignment on its own. Seq.01, a long straight then a turn, loses to LongStream, so the information-adaptive rule is not stronger on simple constant-velocity stretches. NC does not uniformly beat TTT3R. The method is capped by the CUT3R backbone and is not compared with VGGT-style streaming models. Robustness tests swap the middle of a sequence for ImageNet-C corruptions; they are not real onboard weather.