GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation
Jiahao Lu, Minghao Yin, Wenbo Hu, Hengyu Liu, Wang Zhao, Sai-Kit Yeung, Ying Shan, Yuan Liu
cs.CV
2026-09-22
GAE compresses frozen DA3 features into a 64/128-channel latent that one flow decodes to RGB and geometry. FVD drops 12.7% on RealEstate10K; camera error is halved.
Photorealistic frames are cheap. A coherent scene is not. Geometry foundation models can recover depth, cameras, and point maps that agree across a few real views. Video generators still drift: recovered geometry wanders, and the camera misses the requested path. The usual fix hangs extra camera controls, aligned features, or geometry rewards on an appearance latent. GAE treats that as a representation problem. The state the generator evolves does not natively carry 3D structure.
Dumping the perception backbone’s feature pyramid into a flow model is a bad fit. A single DA3-GIANT level has 3,072 channels, an effective rank near 11, and condition numbers from 10^8 to 10^16. Early levels keep correspondence; later levels are smoother and geometrically weaker. Cascades such as GLD model several levels with coupled generators. That preserves readout at a high cost in state size.
Stage 1 sits a learned bottleneck between a frozen DA3 encoder and its frozen DPT geometry head. Four normalized feature levels are concatenated, compressed to a 64- or 128-channel grid that keeps the DA3 patch layout, and decoded back to the full hierarchy in one pass. Freezing the geometry head stops it from hiding information the codec dropped. A separate RGB head reads the same latent, so appearance and geometry must coexist.
Reconstruction says what to keep, not how to arrange it. Token-wise alignment to C-RADIO improves transport smoothness and semantic neighborhoods, then wrecks pairwise spatial structure. Matching posterior similarities to DINOv2 restores those relations. Both teachers are discarded after codec training.
Stage 2 freezes the codec and trains a DiT-style conditional flow on standardized posterior means. Text, metric Plücker rays, and clean reference latents are controls. Only target-view latents move in the ODE. Condition dropout shares one set of weights across text-to-image, camera-controlled video, and reference novel views.
The matched protocol fixes the generator, budget, and sampler: 64 held-out scenes, nine views, 252², one reference, 50 Euler steps, CFG=2.
| Latent | RE10K FVD | DL3DV FVD | RE10K ATE |
| SD-VAE | 258.6 | 373.2 | 0.0072 |
| RAEv2 | 379.4 | 453.4 | 0.0085 |
| DA3 L0 | 298.6 | 376.5 | 0.0085 |
| GAE-64 | 225.7 | 287.0 | 0.0034 |
| GAE-128 | 233.4 | 345.2 | 0.0041 |
Versus the strongest non-GAE controlled latent, FVD falls 12.7% and 23.1%. VGGT trajectory error on RealEstate10K drops 52.8%. Geometry decoded from sampled latents, scored against Pi3 on real frames, has GAE-64 leading all six metrics on DL3DV. The 81-view rollout and text-to-image samples come from a separately trained larger model and stay qualitative.
If a world model must write scenes that perception can read, the generative state should be geometry-native. GAE shows you do not need a feature cascade or a second geometry latent: reparameterize the perception hierarchy into one compact Euclidean code, then run a standard flow. For novel-view and camera-controlled video, swapping the tokenizer may beat scaling the generator first.
The controlled study is low-resolution and short-horizon. Long rollouts and text-to-image have no matched numbers. Geometry I/O is tied to frozen DA3; other backbones and dynamic scenes are untested. Token alignment alone destroys relational structure, so both teachers are load-bearing. GLD and Gen3R are full pipelines, not drop-in latents. Point clouds are unprojected per-frame depths with no cross-view fusion.