RegVGGT: Sustainable Visual Geometry Grounding for Streaming via Regulated Memory
Hongbo Mao, Junjun Jiang, Youyu Chen, Jiaxin Zhang, Zhemeng Dong, Xianming Liu
Harbin Institute of Technology
ECCV 2026
cs.CV
2026-09-20
HIT's training-free RegVGGT keeps the first frame in full and admits the top 1% of tokens per later frame, running thousand-frame streams on an RTX 3090 with stabler pose and reconstruction than fixed-budget baselines.
Feed-forward reconstructors such as VGGT look at every frame at once. As the history grows, the KV cache blows up GPU memory. Temporal causal attention lets the model ingest frames one by one, but keeping every past KV still grows without bound.
Patches split into two camps. Implicit memory compresses history into hidden states and forgets. Explicit memory caps the token count; that budget is wasteful on small scenes and tight on large ones, and nobody can promise that a token dropped now will stay unimportant later. The observation behind this ECCV 2026 paper is simpler: a token's saliency at arrival is a reliable proxy for its later importance.
RegVGGT is a training-free plugin on frozen StreamVGGT. The 24 layers are split by attention shape.
Early and late layers (1–10, 18–24) are nearly query-agnostic; they mostly align later frames to the first-frame world frame. Those layers keep the first frame's KV forever and admit nothing later. The middle seven layers (11–17) are query-specific, sparse, and diagonally structured. Besides the anchor frame, each head keeps only the top γ=1% tokens per incoming frame.
Saliency is scored per head so averaging does not wash out sparse geometric cues. Queries are 2×2 spatially downsampled before column attention is summed. FlashAttention never materializes the full map; the method recovers exact scores from cached log-sum-exp statistics (LSE-Saliency).
Memory size is |M1|+(t-1)·⌊γN⌋, linear at a tiny slope, with no scene-size prior. Baselines are given the same token count RegVGGT actually keeps, so the comparison is about which tokens survive, not about extra memory.
Everything runs on one RTX 3090 (24 GB). Pose is measured on TUM Dynamics and ScanNet at 800/900/1000 frames.
| 1000 frames | TUM ATE (m) | ScanNet ATE (m) | ScanNet RPE rot |
| Evict3R | 0.174 | 1.114 | 15.333 |
| InfiniteVGGT | 0.141 | 1.061 | 11.622 |
| XStreamVGGT | 0.174 | 1.054 | 11.182 |
| RegVGGT | 0.135 | 0.997 | 6.656 |
Point maps on 7-Scenes and NRGBD go out to 300 frames. Evict3R's mean accuracy on 7-Scenes jumps from 0.106 at 200 frames to 0.163 at 300, more than 50%. RegVGGT moves from 0.036 to 0.038. On Bonn video depth, fixed-budget methods stay close on short clips and fall behind as the horizon grows.
Ablations: dropping the global anchor raises ScanNet ATE from 0.165 to 0.175. Dropping the layer split yields 0.176. Per-head retention cuts extra latency from 17.66 ms to 8.29 ms with almost no accuracy change. γ below 0.01 destroys geometry; above 0.1 buys memory, not points. Against the uncompressed backbone, only 5–40-frame clips fit in memory; the 1% tax is negligible there.
For streaming reconstruction on a 24 GB card, the useful rule is: do not freeze a budget, admit tokens per frame, and the admit rate can be tiny. The method never retrains. Evict3R can swap its cap for this regulator; on Bonn at 500 frames, Abs Rel goes from 0.076 to 0.069.
It is the right patch if StreamVGGT is already in production. Under large motion and low temporal overlap, the paper itself raises γ to 0.1.
The layer cut and γ=0.01 are empirical; another backbone may not survive 1%. The fair protocol equalizes token count, so the win is selection quality, not a smaller footprint. The uncompressed backbone can only be compared on short clips, so the geometric cost of 1% at a thousand frames has no direct gold standard. The first frame is a permanent anchor; a blurry or wrong opening view poisons the coordinate frame. Aggressive 1/100 sampling needs a higher γ. Memory still grows linearly. The slope is small, not a hard cap.