Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation
Songlin Yang, Zhe Wang, Xiaotong Zhao, Xi Wang, Ran Zhang, Shuai Li, Taiyi Wu, Songlian Li, Eric Liu, Alan Zhao, Anyi Rao
cs.CV
2026-10-22
On UniVideo's 13B DiT, three FAPL stages make MLLM latents drive generation. Text-only motion order rises from 48% to 61%, and single-factor edit accuracy from 42% to 58%.
Cinematic shots have to be filled in and taken apart at the same time. A script plus a few references does not specify every frame, so the model has to infer how an action continues and how the camera moves. Character, scene, action, and camera also need separate instructions. Changing one should leave the other three where they are.
An MLLM, a multimodal model that reads text and images together, looks like a useful prior. Its hidden states are usually concatenated with a text encoder and VAE reference frames and passed to a Video DiT, a transformer trained with flow matching to synthesize video. The loss only scores the finished clip. The DiT can lower that loss through the text and VAE paths it already knows, and ignore the MLLM latents. A conflict probe makes the bypass visible: when the words and the reference disagree, the picture follows the VAE reference. With MLLM latents alone, Omni-Video 2's 1.3B DiT collapses to noise. The 14B variant and UniVideo's 13B DiT produce artifacts, and can turn one person's continuous action into two people doing different things.
A video-level loss also never says which slice of the condition should move which part of the frame. Left frozen, the MLLM never learns temporal features a generator can use from sparse inputs.
Factor-Aware Predictive Learning (FAPL) adds no modules. It trains UniVideo's 13B DiT in three stages. The MLLM, the connector (the projection from MLLM states into the DiT condition space), and the DiT all start from pretrained weights. Every stage drops the text encoder and the VAE reference, so the only content condition is the MLLM projection. The loss stays flow matching.
Each clip is shown two ways: text alone, or text plus the full target video. Both share that video as the target, and the text-only branch never sees the frames. The edit set has 100K filtered pairs, 25K for each factor, taken mainly from CoinVE-200K, with extra clips and references from the OmniVBench training release. Captions are rewritten into four ordered fields, and a pair changes only one field. Field boundaries are the addresses of the latent spans. Spans need not match in length, and they are not assumed to be disentangled.
Stage I freezes the MLLM and the connector and trains the DiT for 10K steps, with the two observation modes sampled 1:1. Full video teaches the DiT to read temporal detail out of the latents. Text preserves the interface later edits will use. With the side paths closed, the DiT can reduce the loss only by using those features.
Stage II keeps the upstream nets frozen, uses text only, and runs for 15K steps. A pair differs in one factor. That factor's span is cut out of one projected sequence and replaced with the other example's span; the target video is swapped to match, and both directions are trained. Replacement sits after the connector and before the DiT. What gets trained is the link between that local swap and the change-versus-keep pattern in the output.
Stage III freezes the DiT. The MLLM and connector are fine-tuned from their original weights for 10K steps, again at a 1:1 mix, on the same shared targets. From text alone, the upstream net has to emit states the fixed generator can turn into the right semantics and motion. When the full video is visible, it still has to pass through the same connector. Gradients flow through the frozen DiT. The generator weights stay put.
The learning rate is 1e-5 with AdamW. Inference uses 50 steps and a classifier-free guidance scale of 5.0. A script and optional references go through the MLLM into the DiT, with no target video required. The original text and VAE conditions can be switched back on as a separate setup. An edit replaces only the target span. Shots in a multi-shot clip are generated independently, with no cross-shot memory.
CineFactor-10K holds 10K held-out single shots, with no source overlap against training. 5K are text-only sparse-to-video. 5K are factor edits, 1,250 per factor. Multi-shot uses the 100 official ShotVerse-Bench test sequences, two to four shots each.
Motion Order scores action sequencing, higher is better. A frozen π3 model recovers camera trajectories; translation error and rotation error in radians should be low. FiVE's ∩-Acc counts an edit only when both the yes/no check and the multiple-choice check pass. Multi-shot also reports Subject, Scene, and FaceSim-Arc.
| Method | Motion Order↑ | Trans. error↓ | Rot. error↓ | ∩-Acc↑ | Subject↑ | Scene↑ | FaceSim-Arc↑ |
| Omni-Video 2 (1.3B) | 33% | 0.1084 | 1.87 | 28% | 0.435 | 0.373 | 0.412 |
| Omni-Video 2 (14B) | 43% | 0.0817 | 1.49 | 37% | 0.476 | 0.423 | 0.468 |
| UniVideo (13B) | 48% | 0.0763 | 1.54 | 42% | 0.462 | 0.427 | 0.451 |
| FAPL | 61% | 0.0586 | 1.02 | 58% | 0.516 | 0.471 | 0.523 |
Against the same UniVideo 13B, motion order moves from 48% to 61%, translation error from 0.0763 to 0.0586, rotation from 1.54 to 1.02 radians, ∩-Acc from 42% to 58%, and FaceSim-Arc from 0.451 to 0.523. Omni-Video 2 at 14B is lower on every column of this table.
Ablations stay on that 13B model. After Stage I alone: motion order 54%, translation 0.0708, rotation 1.38, ∩-Acc 44%. The features are being used. Editing barely moves.
Stage II keeps the pairs and the step budget fixed, and both runs continue from Stage I. With span replacement, ∩-Acc is 57%; without it, 48%. Motion order is 53% versus 54%, and translation error is 0.0671 versus 0.0654. Camera metrics do not come along with the editing gain.
Stage III trained on text only reaches 57%, 0.0629, 1.26, and 56% on motion order, translation, rotation, and ∩-Acc. The 1:1 mix reaches 61%, 0.0586, 1.02, and 58%. Full-video observations mainly help action order and camera control when inference is text-only.
In the text-only examples the baseline shortens long hair, drops a specified hand gesture, or holds the camera still on a requested 30-degree arc. The full numeric comparison is reported only for UniVideo 13B. On Omni-Video 2 at 1.3B and 14B, the paper only shows that Stage I stops MLLM-only conditioning from collapsing into noise.
Turning off the text encoder and the VAE, then training only the DiT, is what gets MLLM latents into the generation loop. No new block is added. The later span edits and the MLLM update both depend on that interface.
Factor control does not need a disentangled representation first. The four caption fields supply addresses. Paired videos state what changes and what stays. The signal is a span swap. Same pairs, same budget, and dropping the swap drops ∩-Acc from 57% to 48%.
The generator is then frozen and the MLLM is fit to it. Prediction from sparse inputs and reconstruction from the full video share one target, and the supervision remains the generation loss.
A video DiT that already takes MLLM conditions can run these three stages as its training recipe. Scripted generation, single-factor edits, and multi-shot prompting share one condition path. Multi-shot consistency comes from shared references, and there is no cross-shot memory. The gains over UniVideo are measured, and they are still a training improvement on the same backbone. Fidelity and believable acting stay limited by that backbone.
The paper puts the ceiling on the base models. Current MLLMs are weak at reading 3D blockouts, rough layouts built from simple geometry, and weak at turning complex camera moves into controls a DiT will follow. The pretrained DiT limits visual fidelity and hard factors such as realistic acting.
There is no control that spends the same 35K steps on the same 100K pairs while keeping ordinary text-plus-VAE conditioning. Stage II does lock data and budget, so the ∩-Acc move from 48% to 57% can be credited to span replacement. The motion-order move from 48% to 61% is still mixed with extra training. The three stages are not isolated cleanly on that metric.
The main table also does not say which condition paths were open at inference. Sparse-to-video is text-only. Factor editing runs each model through its own standard interface. Higher task scores do not, on their own, show that MLLM latents are sufficient alone.
The edit pairs are selected and generated, then filtered. No human audit is reported for whether only one factor actually changed. Every reported metric is automatic. Multi-shot covers 100 sequences and generates each shot on its own, so it measures reference consistency, not narrative memory. How much meaning leaks across a span cut at a field boundary is not quantified.