RGBX-Next: Towards Realistic Generative Rendering from G-Buffers
Zheng Zeng, Marco Salvi, Lifan Wu, Jan Novák, Daqi Lin, Saeed Hadadan, Yichen Sheng, Robert Pottorff, Shiqiu Liu, Ravi Ramamoorthi, Ling-Qi Yan, Miloš Hašan
cs.CV, cs.GR
2026-08-14
NVIDIA finetunes the Wan 2.1 14B video DiT into a renderer: G-buffers in, photoreal frames out, plus inverse rendering of real video; beats prior methods on most metrics.
Path tracing offers precise control: geometry, materials, and lighting are defined exactly, and the renderer obeys. The cost is asset production that only large studios can afford. Diffusion models are the mirror image: near-photographic realism from a text prompt, but no precise handle on where the camera sits or what the tabletop is made of, and temporal stability is not guaranteed. The bet in this paper is to take half of each: render cheap G-buffers (albedo, normals, depth, material, irradiance, direct lighting) with a traditional pipeline, then hand them to a finetuned video diffusion model that paints in the realism. The same framework runs in reverse, estimating G-buffers from real images and video.
The base is Wan 2.1 14B, a video diffusion transformer (DiT), with the VAE frozen. The core move repurposes the DiT's frame budget into conditioning channels:
The channel-wise concatenation of DiffusionRenderer is the main comparison, and it converges clearly worse, because the model has to force inter-frame attention patterns into attention between input and output signals. VACE-style context tokens were also compared, with lower generality and quality.
Training uses two datasets: an internal set of 900 path-traced video sequences (about 100 frames each, with ground-truth G-buffers), and 6,622 real Pexels videos pseudo-labeled with G-buffers from their own RGB-to-X model, 371 held out for testing, captioned by Qwen2.5-VL. Guidance dropout during training (50% whole-modality dropout, plus spatial masking and Gaussian blur) is what lets you control geometry without lighting at inference. A streaming variant adds clean long-context reference tokens and self forcing, supporting unbounded bidirectional streams.
RGB-to-X on the Hypersim test set (this model was not trained on Hypersim; RGB↔X was):
| Method | Albedo PSNR | Normal PSNR | Irradiance PSNR | Depth PSNR |
| RGB↔X | 18.39 | 19.25 | 11.66 | n/a |
| DiffusionRenderer | 15.29 | 21.54 | n/a | 17.72 |
| RGBX-Next | 20.17 | 21.22 | 25.19 | 29.47 |
For X-to-RGB on the held-out real-video test set, FID comes out at 45.39 for RGBX-Next, against 62.29 for VACE, 62.39 for RGB↔X, and 88.60 for plain channel-wise concatenation. Speed is the weak point: on one A100 80GB, a 17-frame 1280×720 clip takes about 200 seconds for RGB-to-X (CFG=1) and about 400 seconds for X-to-RGB (CFG=3), at 20 sampling steps.
For games and film, the implication is that assets only need to be specified down to G-buffer level, with realism delegated to a learned prior, cutting the cost of high-fidelity material and lighting work. The authors explicitly frame their models as teachers, to be distilled (DMD-style) toward interactive speed, and bet that growing neural inference capacity on future GPUs will let the distilled model overtake Monte Carlo rendering. The inverse direction, video intrinsic decomposition, is a ready-made tool for relighting and compositing. As a method contribution, it is a general recipe for turning any video DiT into a multi-input, multi-output, multi-modality model, with all models promised for release.
From the authors: the models are no faster than the Wan 2.1 14B base, far from interactive; whether G-buffers are the optimal guidance signal is unsettled; no public dataset has reliable roughness and metallicity ground truth, so material estimation is not quantitatively evaluated; streaming error accumulation is mitigated, not removed; and the pseudo-G-buffers in real training data inherit bias from their own RGB-to-X model.
Reading it closely raises more: the internal 900-sequence synthetic set is not published with the paper, so reproduction hinges on the promised model release; the comparison is not a clean sweep, with normal PSNR below DiffusionRenderer and irradiance LPIPS worse than RGB↔X; FID is computed on a held-out set annotated by their own model, making the evaluation partly self-referential; and video and streaming results are shown qualitatively with timing numbers, without independent quantitative video metrics.