ZJU & HKU open-source LayerRecall: a 1.65M-param memory router fixes long-video forgetting
青稞AI · wechat · 2026-09-11
Autoregressive video models squeeze early identity and scene cues out of their KV cache, so characters who leave the frame often come back with a different face or outfit. Zhejiang University (Yi Yang's team) and HKU open-sourced LayerRecall, which trains only 1.65M routing parameters (0.033% of the frozen 5B backbone) to decide what to retrieve and where in the network to inject it.
Key ideas
- What to Retrieve: compact chunk summaries are used only for retrieval scoring; the selected chunks' full K/V are fed to attention.
- Where to Use: layer-wise analysis shows DiT layers differ sharply in long-range attention; memory goes only into a fixed set of memory-sensitive layers. Injecting into all layers (All-Layer Routing) increases abrupt jumps — LayerRecall cuts 2–12Hz high-frequency power ratio from 0.60 to 0.38.
- CHPM training: a 384-frame-history teacher supervises a 32-frame-budget student via denoising predictions, requiring no memory labels.
Results: on 100 held-out three-shot prompts, MemoBench Overall 0.548 and MovieBench 0.578 beat the best baselines (0.531 / 0.548) with VBench-Long (0.978) unchanged vs backbone. The router transfers without retraining to LongLive (0.4985→0.5430) and Self-Forcing (0.3270→0.4965). Extra routing overhead is only 3.5s.
Training/inference code, CHPM pipeline and the 100-prompt eval bank are open-sourced; users must supply Wan2.2-TI2V-5B and LongLive-2.0 weights, and the default 64s/24FPS config needs two GPUs.
More from Multimodal
- Magnific teases October 28 release: 'We're making it move' — cuenca · 2026-10-27
- Midjourney --sref demo: retro Americana Batman art series — gizakdag · 2026-10-27
- AI video is a new medium, not cinema or animation — and it's perfect for depicting dreams — Kyrannio · 2026-10-27
- Dad turns his toddler's 31-year-old quote into a song via SUNO kids' choir — NathanpmYoung · 2026-10-27
- Band's local-AI music video hits 200k views: 900 generations kept 64, 3-second clips beat face drift — Hwang0500 · 2026-10-27
- Adobe Firefly video editor walkthrough: one-sentence Quick Cut turns 32s into 20s — LudovicCreator · 2026-10-26