ZJU & HKU open-source LayerRecall: a 1.65M-param memory router fixes long-video forgetting
青稞AI · wechat · 2026-09-11
Autoregressive video models squeeze early identity and scene cues out of their KV cache, so characters who leave the frame often come back with a different face or outfit. Zhejiang University (Yi Yang's team) and HKU open-sourced LayerRecall, which trains only 1.65M routing parameters (0.033% of the frozen 5B backbone) to decide what to retrieve and where in the network to inject it.
Key ideas
- What to Retrieve: compact chunk summaries are used only for retrieval scoring; the selected chunks' full K/V are fed to attention.
- Where to Use: layer-wise analysis shows DiT layers differ sharply in long-range attention; memory goes only into a fixed set of memory-sensitive layers. Injecting into all layers (All-Layer Routing) increases abrupt jumps — LayerRecall cuts 2–12Hz high-frequency power ratio from 0.60 to 0.38.
- CHPM training: a 384-frame-history teacher supervises a 32-frame-budget student via denoising predictions, requiring no memory labels.
Results: on 100 held-out three-shot prompts, MemoBench Overall 0.548 and MovieBench 0.578 beat the best baselines (0.531 / 0.548) with VBench-Long (0.978) unchanged vs backbone. The router transfers without retraining to LongLive (0.4985→0.5430) and Self-Forcing (0.3270→0.4965). Extra routing overhead is only 3.5s.
Training/inference code, CHPM pipeline and the 100-prompt eval bank are open-sourced; users must supply Wan2.2-TI2V-5B and LongLive-2.0 weights, and the default 64s/24FPS config needs two GPUs.
More from Multimodal
- Creator makes 2D electro-pop anime music video with just a prompt using MiniMax H3 — Hailuo_AI · 2026-09-11
- Street View to driving footage: GPT Astra fetches images, MiniMax H3 turns them into dashcam video — Hailuo_AI · 2026-09-11
- Single-author ECCV 2026 paper makes rolling shutter correction practical — ducha_aiki · 2026-09-11
- AI digital human covers Japanese classic so realistically viewers can't tell — JourneymanChina · 2026-09-11
- ComfyUI Style Explorer Adds LoRA Preview Catalog and Sharing — neonsparksuk · 2026-09-11
- 4 favorite Midjourney V6.1 --sref style codes, ready to copy — michaelrabone · 2026-09-11