GEAR Uses Geometry as Token Addresses, Cutting ATE 80.3% vs Best Baseline

Geometry as Address: Routing Attention to Visual Memory for Long-Horizon Camera-Controlled Video Generation

Zesong Yang, Weikai Chen, Liyuan Cui, Lutao Jiang, Runze Zhang, Yingda Yin, Xiaoyang Huang, Kai Yan, Keyang Luo, Wangguandong Zheng, Xin Wang, Hujun Bao, Zhaopeng Cui

cs.CV

2026-09-28

GEAR addresses visual memory with per-frame geometry and stores history as frame latents. DL3DV ATE is 0.0436, 80.3% below the best baseline, for minute-long camera video.

What problem this solves

Long-horizon camera-controlled video is a memory lookup problem. After the camera leaves a region and later returns, the model has to recover what it already saw from a growing history. Inventing the region from the current clip is the failure mode.

Two standard designs break in opposite ways. Implicit memory keeps every historical frame as tokens and searches with dense attention. The pixels are still there, but only a few patches matter for the current view, and the rest dilutes retrieval as the bank grows. Explicit 3D memory fuses observations into a global mesh, point cloud, or voxel grid and reprojects it. Spatial addressing is then cheap, yet local geometry errors become permanent after fusion and re-enter the next chunk as noisy renders. Lyra 2.0 warps source coordinates and depth into the target view and injects them as DiT embeddings; UCM warps positional encodings. Geometry still selects or aligns a condition. The actual visual tokens remain one hop away.

GEAR splits addressing from readout. Geometry answers which historical cell to read. Attention decides what to copy.

Method

GEAR (Geometry-Enabled Attention Routing) comes from Zhejiang University CAD&CG, HKUST(GZ), and LIGHTSPEED. The backbone is frozen Wan2.1-I2V-14B. Training adds rank-32 LoRA plus Geometric Correspondence Attention (GCA) in every even DiT block. GCA has 262M parameters, 1.9% of the backbone.

Four moving parts:

• Each frame is encoded independently by the VAE and stored as a latent plus camera. There is no globally fused 3D scene. The VAE downsamples spatially by about 16x, so a latent token can cover several surfaces. The local triangle mesh is built at image resolution and only then rasterized onto the latent grid.
• The same source mesh is rasterized under the source camera and the target camera. Shared face IDs become token-level matches. One target patch can keep several historical sources. Geometry mistakes stay local to that lookup.
• Projectable is not the same as visible. An Invisible Octree accumulates free-space and occlusion evidence and stores no appearance. It drops matches that sit behind the current visibility boundary. Updates are conservative: new frames only refine unknown or previously invisible cells.
• GCA runs sparse cross-attention from each noisy target token onto its matched historical tokens, then residual-adds the result. Empty match sets return zero, so unseen regions fall back to the pretrained prior.

From iteration 8K, history-to-video samples get degraded-history with 40% probability: historical latents are noised with τ up to 0.3 and stepped once through reverse flow, so training sees the dirty memory of autoregressive rollout. I2V versus H2V sampling is 30%/70%. At inference, keyframes are chosen by greedy geometric coverage, always keeping the first and latest frames. After each chunk, frozen Depth Anything 3 estimates depth and pose and updates the octree.

Results

Training uses DL3DV-10K at 480×832 with 55-frame clips, AdamW on 32 GPUs, learning rate 1e-4, 10K steps. DL3DV-Evaluation rolls out along ground-truth cameras. WorldScore-Static uses 50 scenes with closed-loop paths for revisits. Camera metrics come from ViPE poses aligned to GT by Umeyama.

ATE is 80.3% lower than WorldStereo's 0.2212, the strongest baseline on that metric. TransErr is 0.0116 versus Lyra2's 0.0157; RotErr is 0.1228 versus WorldStereo's 0.1717. On WorldScore, content alignment is 0.7423 versus 0.7003 for WorldStereo; revisit LPIPS is 0.2019 versus 0.2193. FVD is 837.59 against WorldStereo's 846.46, essentially tied.

Ablations pin the gain on sparse addressing. Removing GCA drops SSIM to 0.1324 and raises ATE to 0.3471. Keeping GCA's parameter count but attending densely over all retrieved history still leaves ATE at 0.2924. Dropping the Invisible Octree raises ATE from 0.0436 to 0.0729. Perturbing correspondences (keep 2 to 3 of 9 matches, shift the rest by 64 to 128 pixels) leaves ATE at 0.0464; GCA down-weights feature-inconsistent distractors.

Sparse geometric addressing is worth more on this table than extra attention capacity.

Why it matters

For camera-controlled world models, this is a usable split: geometry as a pointer, appearance as content. Teams already paying for fusion drift can try per-frame local meshes plus sparse correspondence attention without waiting for a clean global reconstruction. Adaptation cost is modest. The 14B backbone stays frozen; only LoRA plus a 1.9% GCA head are trained.

The method still needs usable monocular depth and cameras. What is shown is single-image plus text, user-specified trajectories, minute-scale roaming. It improves memory access. It does not replace the video prior.

Limitations

The paper is explicit: depth comes from an external 3D model, with extra compute, and joint learning of geometry and addressing is left as future work.

Camera scores are computed from ViPE on generated video. A lower ATE can mix two effects: tighter camera follow, and frames that ViPE can pose more easily. Subjective Quality on WorldScore sits near 0.50 for almost every method; GEAR and HY-WorldPlay both report 0.5018, so that column does not rank anything. Train and the main table are both DL3DV; minute-long out-of-domain claims rest on qualitative figures. The octree never overwrites established evidence, so a wrong visibility cell could persist, and that failure is not measured. Dynamic and non-rigid scenes are not in the protocol.

Terms

• GCA: sparse cross-attention over geometrically matched historical tokens, residual-injected during denoising.
• Invisible Octree: a visibility-only sparse octree that rejects occluded but projectable matches.
• Geometry as address: geometry chooses where to read history; it is not stored as a persistent 3D scene.
• Revisit consistency: recovering previously seen content after the camera leaves and returns.
• Degraded-history: training noise on historical latents to mimic dirty memory at rollout.

Source

• arXiv 2609.34722v1
• PDF
• Hugging Face
• GitHub · https://github.com/zju3dv/geometry-as-address

Related papers

• Same-time reallocation lifts HunyuanVideo sparse attention by 11.8 dB PSNR
• Frozen ALPS tokens hit 85.5% on LIBERO while matching V-JEPA 2-L
• Training-free segmentation: a frozen MLLM draws B-spline masks itself, 71.5 cIoU on ReasonSeg
• Wrong tap, same login screen: LGWM hits 0.987 AUC on GUI transitions in 17 ms
• Asymmetric distillation lifts 10-turn consistency from 3.03 to 3.85
• Best VLM maps scenes at 41.71% mean IoU; only 3.25% clear IoU 0.9

All paper explainers