HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enhancement
Yiyang Cai, Nan Chen, Rongchang Xie, Junwen Pan, Chunyang Jiang, Cheng Chen, Wen Zhou, Zhenbang Sun, Wei Xue, Wenhan Luo, Yike Guo
cs.CV
2026-07-21
HOMIE injects MLLM reference semantics into self-attention via global multimodal guidance, lifting abstract-logo OCR accuracy to 0.452 against SkyReels-V3's 0.326 on a self-curated benchmark.
Subject-driven video generation has a hard sub-task: human-object centric video personalization (HOCVP), where you give reference images and get a video of a specific person interacting with a specific object. Two things must hold at once: the person's face and the object's look must match the references, and the interaction must match the prompt. Prior methods trade these off. Push fidelity and the interaction turns to paste-and-copy artifacts; push interaction and identity drifts. Abstract references like logos are worse, because the model has to reason about a concept rather than copy pixels. When you hand the model several references of one subject (multi-view, OCR text images), old methods have no way to read the latent correspondence between them.
Bringing in a multimodal LLM (MLLM) to read the references is natural, but old recipes either align MLLM features back into the UmT5 text space, which weakens the text encoder's controllability and funnels rich features through a bottleneck, or replace the text encoder outright, which forces a costly re-alignment. HOMIE's pitch is an integration that keeps the text encoder in charge and avoids re-alignment.
HOMIE is built on Wan2.1-14B and Wan2.2-14B video diffusion transformers. References go through an MLLM (Qwen3-VL-2B-Thinking) to pull semantic features, then an MLP, then get concatenated with video and reference tokens into the DiT.
Two designs carry the weight. Global Multimodal Guidance (GMG) injects MLLM semantics into self-attention without plain concatenation: it pools the MLLM's query and key along the time axis into a global representation, then uses it to affine-transform the video query and key, Q̃v = (1+γ)Qv + β, with γ and β produced from the MLLM. The MLLM now handles high-level reference relations while the text encoder keeps general scene control, and the two no longer fight. Modality-Reference Embedding (MRE) is two learned embedding sets: one tags whether a token came from video, a reference, or the MLLM; the other encodes reference grouping, so multi-view references of one subject share an embedding while different subjects get different ones.
Evaluation uses the authors' own 200 samples (100 inter-subject including 25 abstract logos; 60 OCR; 40 multi-view), with no public standard set.
| Method | Face-Sim | Obj-Sim | OCR Acc |
| HOMIE (Wan2.1-14B) | 0.786 | 0.891 | 0.452 |
| Phantom | 0.801 | - | - |
| SkyReels-V3 | - | 0.847 | 0.326 |
| Kling 1.6 | 0.678 | - | - |
On OCR accuracy HOMIE beats SkyReels-V3 by nearly 13 points, and OCR is the hard metric for abstract text and logo references. In the ablation, dropping GMG cuts Face-Sim from 0.786 to 0.697; dropping MRE cuts OCR to 0.376. Each module earns its keep. Training cost is on the order of 10k A100 GPU-hours.
Video generation is moving from "make it move" to "control these exact people and objects," and HOMIE's MLLM integration is an engineering answer that keeps the text encoder and stays cheap. For teams building personalized character video or e-commerce assets, it shows a clean way to decouple semantic understanding from image synthesis.
The biggest gap is evaluation: only 200 self-built samples, no public benchmark, no human-preference study. The paper discusses limitations in an appendix but the body does not spell them out. From the method, the whole pipeline leans on the MLLM reading the reference correctly; if it misreads an abstract logo the downstream generation fails too, and that dependency is not stress-tested separately. An OCR accuracy of 0.452 is also not high in absolute terms, which confirms abstract references stay hard.