The Past Frames the Future: Memory for Autoregressive Video Generation
Harold Haodong Chen, Rongjin Guo, Disen Lan, Wen-Jie Shu, Hongfei Zhang, Hanzhe Hu, Shengtao Yao, Zixin Zhang, Guibin Zhang, Zhefan Rao, Jinxiu Liu, Yexin Liu, Rui Peng, Yuhao Liu, Bin Ren, Shuai Yang, Yukang Chen, Salman Khan, Ying-Cong Chen, Ser-Nam Lim, Rynson W. H. Lau, Nicu Sebe, Yu Cheng, Ming-Hsuan Yang, Qifeng Chen
cs.CV
2026-09-24
HKUST-led survey: AR video memory is history that still steers generation after leaving the local window, cut along forms, functions, operations, learning, and evaluation.
Autoregressive video generation extends a sequence unit by unit: tokens, frames, latents, or short clips. As the rollout grows, full history becomes unaffordable. Attention is quadratic; KV caches, device memory, and latency all cap out. In practice each step sees only the last W units. Identity, layout, motion state, narrative commitments, and earlier world-changing actions often leave that window while they are still needed.
Existing tricks go by many names: rolling windows, KV caches, 3D priors, compressed latents, retrieval banks. Surveys of video diffusion and world models mention long context or retrieval. Few treat memory under a causal outer loop as the design problem. This survey defines memory operationally: historical information kept across outer AR steps that can still steer generation after the original evidence is gone from local context. Dense recent frames fed at every step are active context. History that is selected, compressed, retrieved, or revised and then reused is memory. The split is how history is managed, not how long the window is.
The literature is cut along five axes, then grouped as system characterization, system construction, and evidential validation.
Forms ask what object carries history. Visual memory is RGB or VAE latents still traceable to an observation: high fidelity, high redundancy, and a path for generated artifacts to be written back. Implicit state is native neural state used as memory: managed KV caches, SSM or recurrent states, encoder embeddings. Explicit state has a declared world meaning, such as entity records or point, voxel, or surfel maps. Parametric memory writes experience into adaptable weights, including test-time training (LaCT, RAD-TTT) and temporal LoRA (SlowFast-VGen). Pipelines mix carriers.
Functions ask what must persist. Identity: the same entity is recovered after occlusion or a cut; appearance may change, instance cues may not drift into a different person. Spatial: after the camera leaves and returns, layout and geometry still match; recency is a poor spatial index. Dynamic: motion keeps evolving while off-screen. Semantic: roles, events, and unfinished goals still constrain later shots. Causal: a moved object or an opened door still constrains later frames after the action leaves context. Selective invariance runs through all five: freeze what should stay, allow what should change.
Operations are the lifecycle: write, read, update, manage, integrate. Write policies include admit-all, anchors, utility filters, and event triggers; addressable stores also have to bind the right entity or coordinate. Reads can key on time, pose, content, or a learned query. Under a budget the failure modes rotate among omission, contamination, and misbinding.
Learning is closed-loop because generated frames are written back and errors roll forward. Supervision can hit outputs, memory contents, or state transitions. Training histories range from clean teacher forcing, through corrupted prefixes, to the model's own rollouts. Retention, compression, and retrieval used at deployment need to be present in training.
This is a survey, so there is no new bake-off score. The structured claims are the result.
Memoryless rollouts fail in six recurring ways: entity forgetting, appearance drift, spatial inconsistency, dynamics degradation, semantic drift, and causal reset. A door that recloses a few steps after it was opened is a causal failure, not a quality failure.
On evaluation, visible consistency is not memory evidence. Suites such as VBench show that content is still on screen and generation has not collapsed. They do not show recall after an information gap. Three gap protocols are already in use:
| Protocol | What it queries | Representative benches |
| Entity reappearance | Same instance after leaving and returning | EntityBench, MBench, MemoBench, LiveBench |
| Scene revisitation | Recoverable layout at a comparable pose | MIND, iWorld-Bench, WBench, WorldRoamBench |
| Out-of-view state evolution | The new state after a hidden process | MemoBench, StEvo-Bench, LiveBench |
Goal continuation and delayed action effects still lack clean standard protocols. Most story benches keep the full script in the condition; interactive benches rarely verify that an action succeeded and then query only its later consequence. DINO similarity on adjacent frames measures visible continuity. The same score after a disappear-and-return protocol measures identity memory.
A credible test needs three gates: a real information gap, a check that the queried behavior can be generated when it is visible, and paired-history controls that change the past while holding the current query fixed. Ablating one cache shows that module matters inside one model. It does not replace a cross-model memory protocol.
For long video and world models, a longer window and a larger KV store do not solve "the evidence is no longer in the local condition." The right questions are whether the needed fact was kept, whether the stored state is still correct, whether it is reachable when needed, and whether it actually changes the next step. Those four breaks can look like the same visual drift.
In engineering terms, split the job. Use a window and rolling KV for local continuity. Use updatable reference anchors for identity, with a way to invalidate stale ones. Use pose retrieval or a geometric map for revisits. Use entity and event state for interaction; do not treat an action as a one-step control. If the report is only FVD and VBench, memory skill is buried under image quality.
The survey's own scope is narrow: AR video with persistent cross-step state; short clips without that state are out. World-action models enter the tables only when their memory informs visual generation. The method lists are curated, not exhaustive, and the authors invite additions.
The operational definition reclassifies some speed-oriented KV caches as memory; that border is a judgment call. There is no common experiment that puts LongLive, WorldMem, and LaCT on one information-gap protocol, so carrier trade-offs stay qualitative: visual fidelity is expensive, structured state is editable and lossy, parametric memory is hard to roll back. Delayed-action and goal-continuation protocols are admitted to be immature, yet causal preservation is still listed as a function; the taxonomy runs ahead of the evidence. Self-rollout learning also still lacks a reproducible training recipe.