LayerRecall: A State-Conditioned Memory Router for Long-Horizon Consistency in Video Generation
Yixuan Ding, Jiahao Kong, Wei Huang, Ruijie Quan, Yi Yang
cs.CV
2026-08-28
LayerRecall injects retrieved historical K/V into 10 DiT layers. On 100 multi-shot prompts it leads MemoBench (0.548) and MovieBench (0.578), matching LongLive-2.0 on VBench-Long.
Autoregressive video diffusion rolls out in chunks and keeps only a recent KV window. When a person, object, or attribute leaves the frame, that evidence is evicted. On reappearance the model falls back to the text prompt, and identity, color, count, and layout start to drift.
Prior memory methods expose compressed states, retrieved frames, or longer KV histories. Access is not use. Attention profiling on LongLive-2.0 shows layers disagree about current, recent, and distant context, and the memory-sensitive set is backbone-specific. Two decisions have to be made together: which history to pull, and which layers should see it.
LayerRecall sits on a frozen chunk-autoregressive DiT, LongLive-2.0 with a Wan2.2-TI2V-5B backbone. The physical KV cache uses a sink plus sliding window and holds 80 latent frames (10 chunks).
What to retrieve: each layer pools pre-RoPE keys of a historical chunk into a summary, builds a query from the current chunk's hidden state, and hard-selects up to two chunks by cosine similarity. The forward pass uses the hard choice; training routes gradients through a temperature-soft mixture as a straight-through estimator.
Where to inject: retrieved K/V enter only ten profiled layers, indices [4, 9, 10, 12, 13, 15, 16, 17, 18, 26]. The other twenty layers keep the original sink-and-window path. At memory-sensitive layers the visible budget is an 8-frame sink, up to 16 retrieved frames, and the current 8-frame chunk (32 frames total).
Cross-Horizon Prediction Matching (CHPM) trains the router. Teacher and student share the frozen backbone. The teacher attends over the full 384-frame prefix; the student uses the bounded cache plus LayerRecall. At supervised anchors they match denoised-latent predictions. No long-horizon video targets and no memory-allocation labels are required. The router has about 1.65 million parameters, trained for one epoch on 16 H100s (about 128 GPU hours) over 1,600 multi-shot prompts. Evaluation uses a separate 100 three-shot prompts.
On videos from those 100 prompts:
| Method | VBench-Long | MemoBench | MovieBench |
| LongLive-2.0 | 0.978 | 0.513 | 0.546 |
| MemFlow | 0.981 | 0.531 | 0.542 |
| SkyReels-V2 | 0.992 | 0.466 | 0.508 |
| LayerRecall | 0.978 | 0.548 | 0.578 |
Memory scores lead the table; local continuity matches the backbone. Object reappearance on MemoBench is 0.571 versus 0.519 for LongLive-2.0. Injecting history into every layer hurts temporal consistency: adjacent-frame DINO rises from 0.8795 (all-layer) to 0.9212, and the high-frequency frame-change ratio falls from 0.60 to 0.38. CHPM lifts MemoBench from 0.519 (random router init) to 0.548. A matched ablation of profiled versus random ten-layer sets moves overall from 0.538 to 0.570. End-to-end time on H100 goes from 305.9 s to 309.4 s per video.
The same trained router, with no extra optimization, raises MemoBench on LongLive and Self-Forcing 1.3B backbones; swapping in a self-profiled layer policy gives the best overall. Qualitatively, a mismatched garment in shot 3 can snap back to the shot-1 color without resetting motion or scene layout.
Long-horizon samplers usually buy length by forgetting. LayerRecall is a small, frozen-backbone overlay: retrieve from the current state, inject only where the DiT actually uses distant context. For multi-shot identity and object return, that is cheaper than stretching KV without bound. Retrieval transfers across the tested backbones; the layer allowlist does not, and has to be reprofiled.
This is an incremental systems fix, not a new generator. Gains sit on memory diagnostics. SkyReels-V2 still wins raw VBench-Long.
The layer policy is frozen after profiling and does not adapt per prompt. The authors describe cross-backbone results as portability on the tested models, not universal zero-shot reuse. Random-10 beats the profiled set on identity (0.507 vs 0.495). Evaluation is 100 three-shot prompts; training is 200 optimizer steps. CHPM matches predictions, not teacher attention, so the router approximates behavior rather than a labeled memory map. Self-correction is qualitative only.