RealMe: Immersive Volumetric Video from Minimal Captures
Chenxu Zhou, Zhiyuan Yu, Yudong Jin, Yinji Shentu, Yu Zhang, Jiaming Sun, Xiaowei Zhou
SIGGRAPH Asia 2026 Conference Pa
cs.CV
2026-10-22
RealMe adapts FLUX into multi-view diffusion so eight 720p captures reconstruct 2K volumetric video. On RealVV, novel-view PSNR rises 8.22 dB over FTGS, from 19.05 to 27.27.
Volumetric video is free-viewpoint rendering of a moving scene. Optimizers such as FTGS fit 4D Gaussians directly, and they hold up when the rig is dense. Sparse 720p input leaves unseen surfaces under-constrained, so novel views soften and appearance drifts in time.
Generative fill-in is how recent work loosens capture. Two failures remain. Per-view repair ignores the camera layout, viewpoints disagree, and a generate-then-reconstruct stack passes those errors on. Spatio-temporal attention helps consistency, but common video diffusion backbones often stay below 1K. RealMe, from Zhejiang University's CAD&CG lab with 4DV.ai, aims at one target: 2K volumetric video from a few low-resolution views, close to what a dense rig would give.
The input videos are first fit with a coarse 4D Gaussian scene using FTGS. Geometry is roughly there. Detail is not. Poses are interpolated along the fitted camera path, and the coarse model renders novel-view video as multi-view geometric guidance. An offline detector crops person regions from the originals so local detail is kept. The two condition types are enhanced separately, because frames with different spatial coverage should not be packed into one condition. Both are decoded at a shared 2K resolution and sent back to reconstruction.
The backbone is FLUX.1 [dev], 14B parameters. Typical video diffusion models do not natively reach the 2K this pipeline wants, so the base is a high-resolution image model. Denoising each frame and each camera on its own lets appearance drift in time and geometry slip across views. Only a ControlNet branch and LoRA on selected DiT blocks are trained. The VAE, text encoder, and remaining pretrained weights stay frozen. Condition latents and text features enter every DiT block as residual control.
Stock image position ids have three axes. Tokens that share a latent-grid cell but come from different frames or cameras would get the same code. RealMe extends the id to (t, s, u, v): frame, camera, and the two grid axes, with RoPE on each axis.
Attention follows that layout. Temporal attention stays inside one camera across frames. Spatial attention stays inside one timestamp across cameras. The spatial axis indexes cameras, not patches. Ablations show the factorized pattern beats full attention on visual quality and on downstream 4D consistency.
Training is flow matching. The supervised velocity is the noise latent minus the clean latent. A latent frequency loss with weight 0.1 compares FFT magnitudes: low frequencies stay close to the real capture, high frequencies are asked to bring detail back. The same loss in pixel space runs out of memory at 2K, so the supervision stays in latent space.
RealVV supplies the training pairs: 100 sequences, 20 actors, 48 to 80 views, 300 frames per view, over 1.4 million images. Evaluation sequences are held out. Degradations cover three cases: renders from sparse or deliberately underfit 4DGS, crop and downsample, and Real-ESRGAN-style noise, compression, defocus, and motion blur.
Final reconstruction is still FTGS, supervised by 8 original views plus 40 enhanced views per sequence. Adaptation ran 96 hours on 8 H100s, effective batch size 8, learning rate 10^{-4}. Targets are 2560×1440. Conditions are resized to 1280×720.
Held-out RealVV has 7 sequences, 48 views, 180 frames, at 2560×1440, with 8 spatially spread input views. DNA-Rendering zero-shot uses 4 sequences, 120 frames per view, at 1024×1024.
| Method | RealVV PSNR/SSIM/LPIPS | DNA-Rendering PSNR/SSIM/LPIPS |
| FTGS | 19.0491/0.8142/0.2492 | 20.6439/0.7986/0.2158 |
| DIFIX3D+ | 22.6373/0.8267/0.2890 | 21.9532/0.7920/0.1666 |
| Diffuman4D | 24.4387/0.9113/0.2105 | 23.2704/0.7822/0.1427 |
| RealMe | 27.2706/0.9393/0.1621 | 25.5956/0.8979/0.1261 |
On RealVV, RealMe gains 8.22 dB PSNR over FTGS and 2.83 dB over Diffuman4D. LPIPS falls from 0.2105 to 0.1621. On DNA-Rendering the gain over Diffuman4D is 2.33 dB. FTGS cannot constrain unseen regions. DIFIX3D+ lands at LPIPS 0.2890 on RealVV, worse than FTGS at 0.2492, so scene appearance is not held. Diffuman4D does not build backgrounds and generalizes poorly to long-focus close-ups.
With four cameras, 4C4D scores 19.0405/0.3716 (PSNR/LPIPS), 4K4D scores 15.6413/0.5090, and RealMe scores 24.9304/0.2278.
Consistency is measured by optimizing 4DGS on the enhanced videos plus the captures, then rendering back at those cameras. Diffuman4D reaches PSNRc/LPIPSc of 26.7098/0.1913. RealMe reaches 28.0756/0.1385. Temporal warping error, lower meaning more stable, is 0.00299 for FTGS, 0.00551 for RealMe, 0.03170 for Diffuman4D, and 0.04533 for DIFIX3D+. Among generative methods RealMe has the lowest error, and it is still higher than pure optimization. Heavy smoothing can also shrink TWE.
View counts on RealVV, as PSNR/SSIM/LPIPS: 4 views 24.0490/0.8049/0.2705, 8 views 27.2706/0.9393/0.1621, 16 views 27.3614/0.9159/0.1389. Four to eight views is a large jump. Eight to sixteen adds 0.09 dB of PSNR, while SSIM falls from 0.9393 to 0.9159. Eight views is the default.
Component ablations do not match the main table. The full model there has PSNR 24.8924, against 27.2706 in the main comparison, and the text never says why. Dropping layout-aware position embeddings cuts PSNRc from 27.2688 to 25.2596. Replacing factorized attention with full attention yields PSNR 24.3178 and PSNRc 25.8435. Without the frequency loss, PSNR is 22.6017 and LPIPSc is 0.3113. The latent-space loss reaches 24.9641 and 0.2201 at 73.5 GB, versus 72.4 GB without it. The image-space variant runs out of memory at 2K. Its 1K PSNR of 24.7123 should not be lined up against the 2K numbers.
STAR and FlashVSR appear only in qualitative frames: soft faces or invented texture. RealMe keeps sharper faces, hair, and clothing. No PSNR is reported for that comparison.
For free-viewpoint, telepresence, and VR, cameras are usually the expensive part. An 8 dB PSNR gain over direct optimization, from eight 720p views, means the generative prior fills in some appearance the rig never captured.
The advance is incremental. The scene is still 4D Gaussians, and the reconstructor is still FTGS. Four views to eight is still about 3.22 dB, so missing geometric cues are not filled in cleanly. Adaptation took 8 H100s for 96 hours. Inference still runs a 14B diffusion model plus per-scene optimization. Per-sequence runtime is not reported.
The paper states three limits. High-quality multi-view data is scarce, so large scenes and unusual rigs generalize poorly. If the captures lack geometric cues, coarse 4DGS initialization fails and the rest of the pipeline has nothing to stand on. Scaling the data and speeding up 2K diffusion are left open.
The headline gain is on self-captured RealVV. Zero-shot evidence is 4 DNA-Rendering sequences at 1024, not at 2K. Diffuman4D cannot build backgrounds, so full-scene scores favor RealMe. TWE does not beat FTGS. STAR and FlashVSR have no numbers. Table 2 calls its four-camera scores representative, at PSNR/SSIM 24.9304/0.8735, while the view-count ablation lists 24.0490/0.8049 for four views. That gap is never defined. The pipeline is still generate, then optimize. Error accumulation, which the introduction faults in earlier two-stage systems, is not measured on its own. Training covers 20 actors and human performance. Open scenes are barely tested.