Can 4D Foundation Models Remember?
Guangzhao He, Hadar Averbuch-Elor, Wei-Chiu Ma
cs.CV
2026-09-18
PersistBench uses 360° video as omniscient ground truth. Across 12 4D models, object permanence collapses once the target leaves the input view.
A person who watches a skateboarder leave the frame still has a guess about heading and clothing. Current 4D foundation models, reconstruction or camera-controlled video, are scored mostly on pixel metrics inside views where the subject stays visible. Once the object exits, there is no reference, so “did the model remember” is not a question the benchmark can ask.
PersistBench treats 360° video as an omniscient tape. One panorama yields two perspective trajectories: the input shows the target in the first half and loses it later; the reference keeps the target in view the whole time. The model sees only the input and is asked to render the reference cameras. Out-of-view ground truth now exists.
Visual memory is three scores. Object permanence: SAM2 must track the target in generated frames and a VLM must confirm identity; the score is the fraction of valid frames. Motion continuity: an exponentially decayed, scale-normalized distance between predicted and reference mask centroids, computed only on tracked frames. Appearance preservation: cosine similarity of masked DINOv2 features.
Clips come from 360-1M. ViPE estimates pose, GeoCalib aligns gravity, a VLM finds objects, and SAM3 tracks on a horizontally duplicated equirectangular canvas so wrap-around does not split identities. Camera paths are B-splines optimized with L-BFGS for visibility and smoothness. 24,000 raw clips yield 2,000 pairs, labeled into ten categories and split static versus dynamic.
Twelve public models are probed: reconstruction (CUT3R, 4DGT, CogNVS, NeoVerse), camera-controlled generators (ReCamMaster, TrajectoryCrafter, GEN3C, HyDRA), and video-to-360 systems (ViewPoint, Imagine360, Argus, CubeComposer), the last group cropped back to perspective for a fair interface.
Every model drops once the object is invisible, often from near-ceiling visible scores.
| Model | Static permanence inv. (vis.) | Dynamic permanence inv. (vis.) |
| CUT3R | 3.14% (66.43%) | 0.99% (43.99%) |
| 4DGT | 16.03% (97.00%) | 3.89% (96.21%) |
| GEN3C | 83.97% (95.96%) | 87.06% (96.13%) |
| TrajectoryCrafter | 81.56% (99.55%) | 77.80% (99.74%) |
| Argus | 84.06% (99.25%) | 62.42% (98.97%) |
| Imagine360 | 34.17% (98.24%) | 25.84% (98.14%) |
Models that unproject observations into the target view before inpainting, GEN3C, TrajectoryCrafter, NeoVerse, hold up better out of view. Implicit memory such as CUT3R loses the object almost immediately. Motion continuity is a separate skill: correlation with permanence and appearance is 0.34 and 0.54, while permanence and appearance correlate at 0.93. Static and dynamic scores track each other. Argus leads static permanence and continuity and still drops hard on dynamic clips, matching training data with few movers and simple cameras.
A 4D world model trained mostly on videos where the subject never leaves can look strong on novel-view synthesis and still have no memory. This benchmark turns that complaint into three reference-based numbers. Anyone working on occlusion, reappearance, or long-horizon dynamics can sit on this table. The training hint is specific: curate hide-and-reappear data, and explicit geometric conditioning turns generation into refinement instead of invention.
Dataset construction and scoring all depend on off-the-shelf models. ViPE poses, SAM2 tracks, and the Qwen judge add error that is billed to the system under test. Rankings survive component swaps in the supplement, which means the order is not extremely fragile, not that the scores are clean. Source videos are YouTube 360, so the scene mix follows that corpus. HyDRA only predicts after exit, so it has no visible-segment baseline. Motion continuity is ambiguous when several futures are physically plausible; the supplement discusses this.