Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction
Ziren Gong, Guo Chen, Yongjia Li, Yihua Shao, Fabio Tosi, Stefano Mattoccia, Matteo Poggi, Hao Tang, Fei Ma, Shuyan Li, Ziyang Yan, Nicu Sebe, Ling Shao, Jianfei Cai, Qi Tian, Ming-Hsuan Yang
cs.CV
2026-09-30
A 16-author survey sorts 102 4D reconstruction methods into four NeRF and three Gaussian families, consolidates 22 datasets, and contrasts their speed, fidelity, and geometry.
4D scene reconstruction recovers the geometry, appearance, and motion of scenes that change over time. Since D-NeRF in 2021, the field has grown along two separate lines, implicit radiance fields and explicit Gaussian splatting, and it has outgrown its own vocabulary: the term 'deformation field' names an MLP that warps coordinates into a canonical space in the NeRF line, and a network predicting per-Gaussian attribute offsets in the Gaussian line. Every paper also benchmarks on its own dataset slice, so cross-comparison rarely works. Prior surveys mostly treat dynamics as a chapter inside static neural rendering; the few with more methods lack evaluation tables and a unified taxonomy, and the closest one (Zhu et al.) covers only 52 methods and 10 datasets, with no autonomous driving.
This survey, from 16 authors across Bologna, Peking, Hong Kong Polytechnic, Trento, Monash, and Google DeepMind, fits 102 methods and 22 datasets into one frame and adds cross-verified quantitative tables.
The taxonomy turns on one question: where does time live in the representation?
The NeRF family splits into four classes. Deformation-field methods learn a static NeRF in a canonical space (a reference state of the scene) plus a network mapping a point at time t back into that space, in effect a static template plus a time-varying warp; compact, but a single continuous field breaks under large topology changes, which later work patches with motion-aware subspace decomposition. Implicit 4D primitive methods feed time straight into the network, F(x, d, t) → (c, σ), trading canonical decomposition for flexibility at higher compute cost. 4D feature-volume methods factorize the space-time volume into low-rank structures: HexPlane and K-Planes multiply six planes (three spatial, three spatio-temporal) element-wise to fetch a feature, saving memory and time at the price of blurring high-frequency motion under interpolation. Temporal-prior methods skip explicit time parameterization and add optical flow or physics constraints as regularizers; these stack on top of the other classes.
The Gaussian family splits into three. Explicit 4D primitives give each Gaussian a 4D mean and a 4×4 covariance, then slice it into a 3D Gaussian via the conditional distribution at query time before standard rasterization: strong temporal consistency and real-time rendering, but the highest memory per primitive. Deformation-field methods keep static 3D Gaussians in a canonical space and let an MLP predict per-timestep offsets to position, rotation, and scale while freezing the rest; more parameter-efficient, a natural fit for non-rigid motion and view-dependent specularities, at the cost of MLP inference inside the render loop and occasional jitter. Frame-wise methods optimize an independent Gaussian set per frame with inter-frame priors such as rigidity or optical flow, plus dynamic addition and removal of primitives for topology changes; simple and streaming-friendly, but storage grows linearly with sequence length and long-range coherence stays weak.
| Benchmark (metric) | Top method | Score | Reference |
| Neu3D (PSNR, NeRF) | MSTH | 32.4 | DyNeRF 29.6 |
| Neu3D (PSNR, GS) | FreeTimeGS | 33.19 | 4DGS 32.01 |
| D-NeRF dataset (PSNR) | SLS4D | 34.84 | D-NeRF 30.50 |
| NeRF-DS (PSNR) | EfficientGS | 24.65 | Deformable-3DGS 24.10 |
| nuScenes geometry (RMSE) | OmniRE | 1.89 | Deformable-3DGS 2.97, D-NeRF 7.11 |
Three takeaways. Inside the NeRF family, feature-volume structures dominate: SLS4D reaches 34.84 PSNR on the D-NeRF benchmark against the original D-NeRF's 30.50, a gap over 4 dB, and structured grids beat pure coordinate MLPs on small indoor dynamic scenes. External priors are a reliable source of gains: optical flow (ST-4DGS, CD-3DGS), semantics (Gear-NeRF), and above all LiDAR, with LiDAR-supervised methods cutting nuScenes geometry error sharply; pure-RGB methods struggle at driving scale. The efficiency split is stark: Gaussian methods render at hundreds of FPS through rasterization, with 4DRotorGS at 277, while most NeRF variants stay under 15 and several under 1. The cost is model capacity, which is exactly why compression work such as 4DGC and Ex4DGS exists.
This is a selection map, not a new method. Real-time rendering points to explicit 4D primitives or frame-wise training; non-rigid motion and scene editing to deformation fields; tight memory to feature planes; autonomous driving to LiDAR. The 22 datasets are grouped along five axes (synthetic/real, monocular/multi-view, short/long horizon, human-centric/general, RGB-only/multimodal), so benchmark shopping becomes a table lookup. For anyone entering the area, the mapping from design class to natural habitat replaces weeks of scattered reading, and the companion GitHub list keeps it current.
Stated by the authors: the quantitative tables collect numbers from the original papers, cross-verified but not re-run under one protocol; works without released code or configurations were deprioritized, so coverage is biased toward consistent benchmarks. The metrics section concedes that temporal coherence, topology change, and long-term consistency have no standardized evaluation protocol, while PSNR/SSIM/LPIPS are all frame-wise and measure exactly the dimension 4D reconstruction cares about most. Scope covers per-scene optimization only; feed-forward generalizable models are excluded from the core and appear just as future work. Reading the tables adds concerns: SSIM/LPIPS columns carry many gaps, and the training-time entry for DyNeRF sits orders of magnitude from its neighbors, so cross-table comparison deserves care.