Think Locally, Refine Globally for Memory-Efficient 3D Reconstruction
Jingke Zhou, Chenhang Ma, Zhizhou Zhong, Mingkai Liu, Zhuang Zhou, Yicheng ji, Binghua Su, Bo Cai, Xianliang Huang
cs.CV, cs.AI
2026-09-18
LoG-VGGT runs VGGT in 4-frame windows with cross-window attention at four geometry layers, and refines cameras with register tokens. 7-Scenes Acc@1000: 0.060 vs InfiniteVGGT 0.065.
VGGT dumps every frame's tokens into global attention and emits depth, pose, and point maps in one forward pass. That works on short clips. On long streams the O(N²) memory bill fills the GPU.
Patches split into two camps. StreamVGGT caches every historical key/value pair and eventually OOMs. WinT3R bounds memory with a sliding window, then pipes information across windows through a persistent hidden state; local errors ride that state and the trajectory drifts. InfiniteVGGT caps the cache with rolling memory and per-layer token eviction. Eviction is a heuristic with no explicit cross-window geometry constraint, so drift remains.
The design target is local geometry that is good enough, global pose that does not wander, and memory that does not grow with T.
LoG-VGGT thinks inside a window and refines cameras at the end.
The backbone is overlapping windows of size K=4 with stride 2. Inside a window the model keeps VGGT's frame attention and window-level global attention, so cost stays O(K²) regardless of sequence length. History is not stored in full. Cross-Window Attention runs only at the four geometry layers VGGT already feeds to its DPTHead (blocks 4, 11, 17, 23): the current window's queries attend to the concatenation of current tokens and the previous window's cached key-layer tokens. Overlapping frames are aligned on purpose, and geometric continuity leaks across the seam. Caching four layers keeps memory constant. Those four layers are the ones the depth and pose heads already read, so the cross-window path does not invent a new feature source.
Depth mostly needs local continuity, so the sliding pass is enough. Camera pose still accumulates error. Global Camera Consistency Refinement (GCCR) collects register tokens from every window into a compact set and lets each camera token cross-attend to that set. Poses are jointly corrected, but the compute hits a handful of tokens rather than another full-sequence attention.
Training is two-stage: 100 epochs on 12-frame batches to stabilize local geometry, then 10 epochs on 64-frame batches. The mix is 11 indoor/outdoor, synthetic/real datasets on 16 H20 GPUs. The vision encoder is a frozen DINO. The loss is camera L1 plus uncertainty-weighted depth; there is no explicit point-map term, because 3D points are determined by depth and cameras.
On 7-Scenes, mean Accuracy barely moves with length:
| Method | 200 frames | 500 frames | 1000 frames |
| TTT3R | 0.040 | 0.062 | 0.101 |
| InfiniteVGGT | 0.038 | 0.043 | 0.065 |
| LoG-VGGT | 0.036 | 0.037 | 0.060 |
On NRGBD, Acc at 200 frames is 0.034 versus WinT3R 0.037 and InfiniteVGGT 0.049; at 1000 frames it is 0.091 versus InfiniteVGGT 0.097.
Depth test sets are held out of training. Sintel (50 frames) Abs Rel is 0.321 (InfiniteVGGT 0.323). Bonn (110) is 0.054 (WinT3R 0.058). KITTI (110) is 0.109 (TTT3R 0.113, InfiniteVGGT 0.173). InfiniteVGGT drops hardest on KITTI.
Pose uses 500-frame samples per scene. AUC@30: ScanNet 78.56 (InfiniteVGGT 78.46, TTT3R 71.36); TUM-D 91.17 (90.03 / 86.75); CO3Dv2 88.24 (88.02 / 83.31). The margin over InfiniteVGGT is thin; the ranking is stable. ScanNet and CO3Dv2 sit in the training mix; TUM-D is the cleaner split.
Ablations split the two modules. Drop CWA at 200 frames and the run OOMs. Drop GCCR and 7-Scenes Chamfer rises from 0.032 to 0.045, while CO3Dv2 AUC@30 falls from 88.24 to 44.27. Windowed geometry alone cannot hold long-horizon pose.
GCCR can run periodically on past frames only: every 50 frames reaches AUC@30 63.85 on CO3Dv2; every 100 frames 74.91; a final full-sequence pass 88.24. Accuracy versus latency is a real trade.
On a 1000-frame sequence the CWA backbone takes 83.9 s (99.84% of runtime); full-sequence GCCR is 134 ms. FPS stays 12.4 / 12.3 / 12.3 / 11.9 at 100 / 200 / 500 / 1000 frames. InfiniteVGGT falls from 7.0 to 5.4.
For streaming reconstruction this is a cleaner path than shrinking a KV cache: lock attention inside the window by construction, and spend global compute only on cameras. Full-sequence GCCR is almost free. A production setup can refresh every 50 or 100 frames, or update online; FPS at 1000 frames drops from 11.9 to 8.2 and still beats InfiniteVGGT.
This is not a new geometric representation. It is an engineering scale-up of VGGT. Gains versus InfiniteVGGT are small on short clips. The gap shows up past 500 frames, where pose holds and speed does not fall off.
Depth has no global refinement. The authors say large viewpoint changes, long-range occlusions, and globally ambiguous depth may hurt, and the tables are not uniformly first: Bonn 110-frame δ<1.25 is 0.968 versus InfiniteVGGT 0.972; at 500 frames Bonn δ is 0.952 versus 0.960.
Full-sequence GCCR sees future frames. Strictly online runs must use the periodic or online variant and will give up some accuracy. Register tokens still accumulate, so FPS dips slightly. Window 4 and stride 2 are inherited from WinT3R with no window-size sweep. Low light, textureless regions, reflections, fast motion, and highly dynamic scenes are listed as failure modes without numbers.