VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward
Zhaochong An, Orest Kupyn, Théo Uscidda, Andrea Colaco, Karan Ahuja, Serge Belongie, Mar Gonzalez-Franco, Marta Tintore Gazulla
ECCV 2026
cs.CV
2026-03-28
VGGRPO stitches video-diffusion latents to a geometry foundation model and runs GRPO with camera-smoothness and reprojection rewards in latent space, skipping repeated VAE decoding; on Wan2.1/2.2 it beats Epipolar-DPO and VideoGPA on consistency and smoothness while cutting compute 24.5%.
Large video diffusion models look good but are geometrically unreliable: jittery camera trajectories, drifting scene structure, objects that fail to align in 3D. Two prior fixes both have hard limits. Adding geometry modules to the generator weakens the broad generalization inherited from internet-scale pre-training. Geometry-aware alignment corrects after generation, but existing methods work only on static scenes, and their rewards are computed in RGB pixel space, which forces repeated VAE decoding of the latents and is both expensive and sensitive to decoding noise. The goal: leave the generator's structure untouched, skip the repeated decoding, and still handle dynamic scenes.
The centerpiece is a Latent Geometry Model (LGM). It stitches the video-diffusion latents to a pretrained geometry foundation model through a learned 3D-convolutional connector: specifically, it replaces the first several layers of the geometry model so the latents map directly into its intermediate feature space, with the connector trained by feature-alignment error on calibration data. Geometry can then be read straight from the latents, with no return to pixel space. A deliberate choice is to build the LGM on a geometry model that supports 4D reconstruction (Any4D), which lets it handle dynamic scenes natively and sidesteps the static-only limit of prior work.
On top of this, latent-space GRPO uses two complementary rewards: a camera-motion-smoothness reward computed from scale-normalized acceleration of predicted camera poses to penalize jitter, and a geometry-reprojection-consistency reward that reprojects predicted point clouds across views and compares depths to enforce cross-view coherence. Both rewards live in latent space, so the whole policy update needs no repeated VAE decoding.
On static (DL3DV, RealEstate10K, 190 captions) and dynamic (MiraData, 200 captions) benchmarks, Wan2.1-1B and Wan2.2-5B are fine-tuned with LoRA (group size 64) and compared against Base, SFT, Epipolar-DPO, and VideoGPA. Wan2.1-1B, static:
| Metric | VGGRPO | Epipolar-DPO | VideoGPA |
| Subject consistency | 59.47 | 54.21 | 53.68 |
| Background consistency | 66.84 | 55.79 | 56.32 |
| Motion smoothness | 57.00 | 45.50 | 42.50 |
On Wan2.2-5B dynamic scenes, subject consistency is 62.63 (Epipolar-DPO 52.11, VideoGPA 54.74), motion quality 68.42 vs 58.95/60.53, motion smoothness 56.50 vs 38.00/40.00. For efficiency, latent rewards cut per-step time from 54.73s to 41.33s (24.5% less) and peak memory from 76.80 GB to 68.57 GB. The geometry model matters: on dynamic scenes Any4D (visual quality 59.57, motion quality 67.21) beats VGGT (54.96, 60.61).
For anyone building video generation, the appeal is improving geometry without breaking pre-training. Structure-modifying methods trade away generalization; VGGRPO keeps the internet-pre-trained base intact by doing only post-training with latent-space rewards, cuts alignment compute, and extends to dynamic scenes that prior methods could not handle. The LGM is also a reusable component: swap in a stronger 4D geometry model and the gains continue.
There is no dedicated limitations section; a few issues are visible. Most clearly, Dynamic Degree drops slightly (0.3962 vs 0.4237 baseline): the smoothness reward suppresses camera jitter, and RAFT optical flow misreads "less jittery" as "less dynamic," so smoothness and "looks dynamic enough" are partly opposing metrics. Training the LGM itself costs about 1,536 GPU hours, a non-trivial up-front investment. Performance also depends heavily on the underlying geometry model, so Any4D's quality caps the ceiling. Finally, the paper does not run a subjective study to confirm that better geometric consistency reads as better perceptual quality.