L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory
Jinhyeok Choi, JoungBin Lee, Jongmin Lee, Minkyeong Jeon, Jiyoung Kim, Kaeun Han, Yoonjin Yoon, Seungryong Kim
cs.CV
2026-10-30
Every 3D Gaussian in the scene memory carries a VAE latent, rendered straight into the video DiT's latent space: 9 FPS world generation with the best loop-closure consistency.
Interactive world models generate video chunk by chunk conditioned on user actions. Two things break over hundreds of frames: does the scene still look right when the camera returns to where it has been, and can any of this run in real time. Temporal-context methods (history frames, KV caches) keep memory growing linearly with stream length; in the paper's 400-frame benchmark, WorldKV hits OOM and Yume 1.5 peaks at 66.7 GiB while running at 1.42 FPS. Explicit 3D-memory methods anchor content spatially, but feed-forward variants predict dense primitives for every frame and accumulate them without bound, so the same region gets covered repeatedly and rendering slows down; optimization-based variants stay compact but iterate per scene, far too slow for real time (ArtiFixer: 0.10 FPS). Consistency and throughput pull the memory design in opposite directions.
L3G-World alternates two modules autoregressively: a memory updater (L3G-Mem) and a memory-conditioned video DiT (L3G-Gen). Two decisions carry the paper.
First, memory grows with scene extent, not stream length. L3G-Mem renders the previous Gaussian memory under the current cameras to get opacity and RGB-error maps. Pixels with low opacity (geometry never stored) or high error (stored badly) form a spawn mask; only masked regions are K-means clustered into spawning queries that decode into new Gaussians. Gaussians already visible are retrieved as memory tokens and updated in place through gated residual predictions. Spawn and refine tokens pass through one shared transformer, with a global self-attention layer letting new and old content exchange information. The memory is a scene representation that gets revised, not a buffer that only accumulates, so early geometric errors can be corrected by later observations.
Second, every Gaussian carries a VAE latent. Video diffusion operates in VAE latent space, and prior 3D memories had to render RGB and re-encode through the VAE before conditioning the DiT, which is too expensive inside a real-time loop. L3G stores a 16-dim VAE feature per Gaussian and splats it directly into the target camera's latent space, concatenated channel-wise with the noisy latents. The ablation is steep: without latent lifting, VAE-conditioning throughput drops from 943.6 FPS to 2.97 FPS and the rendered latent quality falls with it (cosine similarity 0.782 vs 0.467).
The stack: Depth Anything 3 as geometry backbone, Cosmos-Predict2.5-2B fine-tuned with LoRA as the generator, then AnyFlow-style flow-map plus on-policy distillation down to 4 denoising steps and 3 latent frames per autoregressive step. That distillation is where the 9 FPS comes from.
Core comparison on DL3DV, using the paper's 481-frame loop-closure protocol (walk out, retrace in reverse, compare frames at matching poses):
| Method | Primitives (K) | Memory (MiB) | VAE render FPS | Loop PSNR | FID |
| Lyra 2.0 (point cloud, leanest baseline) | 1,273 | 92.3 | 1.69 | 13.60 | 52.26 |
| NeoVerse (Gaussian) | 8,766 | 501.6 | 3.11 | 13.53 | 152.25 |
| Spatia | 9,647 | 220.8 | 0.08 | 13.96 | 89.39 |
| L3G-World | 11.9 | 40.1 | 1,086.6 | 16.00 | 40.57 |
Pipeline-level (one H100, 400 frames): 7.4 GiB peak memory and 9.02 FPS, the only method marked real-time; Yume 1.5 sits at 66.7 GiB and 1.42 FPS, NeoVerse at 36.8 GiB and 1.99 FPS. On quality, L3G-World takes the best LPIPS, FID, subject consistency, and background consistency on DL3DV, repeats those four on RealEstate10K, and posts the best loop-closure PSNR there at 18.36. The distilled Fast variant trades quality for speed, with loop PSNR dropping from 16.00 to 14.59. Ablations back both designs: disabling adaptive spawning inflates the Gaussian count by about 1.8x while quality drops, and disabling refinement degrades every metric, direct evidence that accumulate-only 3D memory cannot hold long-horizon consistency.
For teams building world models, the useful claim is architectural: explicit 3D memory does not have to choose between dense-but-redundant and compact-but-slow. Selective spawning plus residual refinement gives O(scene-extent) memory growth at feed-forward speed, and storing the downstream model's own latent inside each primitive removes the encode tax that made 3D-conditioned video generation slow. That second trick transfers to any system feeding an explicit memory into a latent diffusion model. At 9.02 FPS, about 4.5x the next-best method, it is still a third of the 30 FPS smooth interaction needs, but it lands inside a usable interactive range on a single GPU.
The paper concedes that on RealEstate10K several baselines beat it on camera-pose error and MEt3R geometric consistency, and on DL3DV its rotation error is only second-best. Appendix D shows specialized online 3DGS reconstructors (ReCoSplat at 480x480) with higher NVS fidelity; the authors position L3G-Mem as a good-enough memory, not a state-of-the-art reconstructor. Points to hold onto: evaluation resolutions differ across methods (480x832 here vs 704x1280 for Yume 1.5), which softens direct quality comparisons; the 481-frame loop protocol is the authors' own, not a community benchmark; and the abstract's headline multipliers (38.6x fewer primitives, 3.2x faster updates) are computed against "the most efficient baseline per metric" and are hard to reproduce exactly from the tables, so the raw table numbers above are the safer reference. The 9.02 FPS figure comes from an H100 while the main tables use an A100. Training and evaluation both use DL3DV and RealEstate10K, real-world walkthrough scenes, and the only control signal is camera trajectory, so this remains far from a Genie-style interactive world with actions and physics.