FAPL From HKUST and Tencent Makes Video DiTs Actually Use MLLM Latents
tencent · hf · 2026-10-23
Researchers from HKUST MMLab, Tencent Video, and École Polytechnique propose Factor-Aware Predictive Learning (FAPL). MLLM representations hold useful priors for cinematic video generation, but Video Diffusion Transformers often bypass them via native conditioning. FAPL uses latent-only generator grounding—training the DiT with a frozen MLLM and connector—plus factor-aware control signals to enable sparse temporal inference and factor-level cinematic control over character, scene, action, and camera.
More from Multimodal
- 156 AI-painted pumpkins: ChatGPT draws "based on everything it knows about you" — Kyrannio · 2026-10-23
- LoRA Studio preview brings Krea 2 LoRA training to 12GB GPUs, seeks feedback — homebrewedstuff · 2026-10-23
- Oxford-NUS survey unifies video understanding and generation under world modeling lens — Shengqiong Wu · 2026-10-23
- OpenKatana: open-source code-driven video editor inspired by Higgsfield's Katana — rk0neki · 2026-10-23
- Open-source ComfyUI node boosts batch diversity by coupling initial noise, no extra steps — Current-Ad-198 · 2026-10-23
- Fudan's Code2Video-Compass benchmarks 15 MLLMs on code-based video generation across 2,104 tasks — Fudan-University · 2026-10-23