OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing
Xiaoyan Cong, Tianyu Wang, Hyeonho Jeong, Zhifei Zhang, Yuqian Zhou, Yijun Li, Shaoteng Liu, Zhe Lin, Srinath Sridhar, Soo Ye Kim
cs.CV
2026-10-16
OmniXRef edits video from text, image, and video references. On 125 cases its score is 3.77; camera rotation error is 0.874 rad, about 40% of CamCloneMaster.
Camera paths, falling rain, and the timing of a gesture all sit on a timeline. Text rarely pins them down, and a still image has no timeline. A finished shot often needs several references at once: one clip for motion, another for the camera, an image for the subject's appearance.
Editing models have stayed split by modality. EditVerse, VIVA, InstructX, and Kiwi-Edit take text, sometimes plus one image. ReBind and TIDE take several images. CamCloneMaster and RefVFX each take one kind of video reference, camera or effects. Seedance and MiniMax already accept wider mixes, but much of that stack generates a new clip. Editing an existing video still lacked one interface. OmniXRef defines the task on four axes: reference count, modality, attribute (camera, motion, content, style, or VFX), and how text, images, and video are combined.
Each control is a reference bank: a task definition, an edit instruction, and an optional image or video. The task sentence locks what may change. A camera bank may move the trajectory and nothing else. Any number of banks share one interface.
A frozen Qwen3-VL-8B-Instruct reads the system prompt, the source, and every bank. The prompt tells each bank to use only its own reference. A single-stream diffusion transformer, close to EditVerse, replaces the MLLM visual tokens with VAE features and runs full attention over the source, the references, and the noisy target. The loss is flow matching.
Mix-Patchify picks the spatial patch by task. Source, target, and content references stay at (1, 1, 1) so identity and texture stay sharp. Camera, motion, style, and VFX use (1, 2, 2): time is not downsampled, and spatial tokens drop by up to 4x. A uniformly fine grid is harder to learn in the same number of steps, because nothing tells the model which attribute to read.
R²-RoPE changes the asset-index axis of 4D RoPE. Sequential indices only cover asset counts seen in training, so extra references at test time fall outside that range. The noise tokens' index also shifts as references are added, so one index can mean the output in a one-reference sample and a reference bank in a longer one. The system prompt, source, and noise keep fixed slots. Reference banks draw 8 reserved slots at random. Each training sample has at most two visual references, but all eight slots see data.
Training starts at 272p on single-reference edits. The learning rate falls from 1.5e-5 to 1e-5 over the first 5,000 steps. Two-reference samples come next, still at 272p, then everything moves to 480p. Classifier-free guidance drops the instruction and keeps the picture, so the boost applies to the named attribute. Dropping the whole bank would boost the entire reference and pull unrelated pixels through.
OmniXRef-Data has about 10M examples, about 7M of them text-only. Camera is about 1.5M, content and style about 650K each, motion and VFX about 100K each. The only two-reference mixes are content plus camera and motion plus camera, 50K each. Source and target differ mainly in the edited attribute. The reference shares that attribute and shows different content, so copying its pixels is wrong. For effects and style, Qwen3-VL filters about 2M candidates. GPT Image 2, a propagation model, and Wan2.2-TI2V-5B then build the stripped source and a reference in another scene. Camera pairs are Vista4D reshoots of two videos along one path, then crossed. Motion comes from MotionPair-60K. Content uses RefVIE.
OmniXRef-Bench has 125 single-video-reference cases across the five attributes. Gemini 3.8 Flash returns labels, not scores. The composite is 0.45xRA + 0.25xmin(VQ, RA) + 0.30xSP: adherence, visual quality, and source preservation. Capping quality by adherence stops a pretty video that missed the edit from scoring well. Camera error uses VGGT-Omega on 32 aligned frames, with translations normalized per trajectory and summed over the other 31 frames. That measure tracks shape and timing, not absolute distance or zoom.
Source preservation sits above adherence. Leaving the source alone is the easier half. Motion adherence is the weak cell, at 2.86.
| Category | RA | VQ | SP | Score |
| Content | 4.69 | 3.87 | 4.70 | 4.48 |
| Style | 3.71 | 3.60 | 4.65 | 3.74 |
| VFX | 3.56 | 3.25 | 4.52 | 3.55 |
| Camera | 3.48 | 3.74 | 5.00 | 3.81 |
| Motion | 2.86 | 3.88 | 4.57 | 3.27 |
| Average | 3.66 | 3.67 | 4.69 | 3.77 |
Specialist baselines lose on adherence. Their preservation scores stay high. RefVFX scores 1.98, 2.24, and 2.49 on content, style, and VFX, with RA of 0.94, 1.12, and 1.69, while SP stays between 4.39 and 4.90. CamCloneMaster's camera score is 3.20 (RA 2.52, SP 4.99). MotionFollower's motion score is 2.14 (RA 2.02).
| Method | RotErr | TransErr | CamMC |
| OmniXRef | 0.874 | 3.961 | 4.437 |
| CamCloneMaster | 2.086 | 9.329 | 10.332 |
| Seedance 2.0 | 4.443 | 22.455 | 24.008 |
| Seedance 2.5 | 4.373 | 22.445 | 24.001 |
Rotation error is 0.874, about 40% of CamCloneMaster's 2.086 and about one-fifth of Seedance 2.5's 4.373. On the paper's commercial comparison, OmniXRef leads overall Score, RA, and SP. Visual quality still trails, which the paper ties to backbone capacity. Fine-tuning MiniMax-H3 on the same data, with no architecture change, beats the original MiniMax-H3 on all five categories and beats OmniXRef on camera (3.99 vs 3.81) and VFX (3.83 vs 3.55).
Ablation composites: 3.77 full, 3.18 without R²-RoPE, 3.13 without the curriculum, 2.86 without Mix-Patchify, and 2.83 when the whole bank is dropped. R²-RoPE is the smallest composite drop, but mean RA falls from 3.66 to 2.86 while SP stays at 4.50. Preservation is carrying that score. Without Mix-Patchify, motion RA falls from 2.86 to 1.51.
With the instruction fixed, two sunset references produce two different lighting results. At test time the model can stack three video references, motion plus camera plus style, past the two-reference training cap. Seedance 2.5 largely misses the motion and the camera. On one A100, a 480p, 121-frame clip costs 30.5 seconds per denoising step in the fine single-reference setting. Four references at patch (1, 1, 1) raise latency to about 3.7x.
Text editing survives. VEFX-Bench GeoAgg is 2.708, above UniVideo (2.516), VACE (1.775), and Wan 2.6 (2.146), near Grok Imagine (2.723) and Seedance 2.0 (2.766), and below Kling o3 Omni (3.057). On ReCo-Bench the Add total is 9.08, the highest listed. Replace is 8.92, just under Mamoda2.5 at 9.06. Remove is 7.51 against 8.67. Style is 8.52 against ReCo at 9.17.
The portable pieces are the reference bank and the synthetic triplets. The Adobe backbone can be swapped. A task sentence separates which attribute to change from what the reference looks like, which is what keeps several conditions from bleeding together. MiniMax-H3, fine-tuned on this data with no new modules, already passes OmniXRef on camera and VFX.
Against commercial editors the gain is adherence and preservation. Motion RA of 2.86 and VFX visual quality of 3.25 are not a finishing tool. At 480p and about 30 seconds a step, the runtime is still a lab setting.
The paper states four gaps. Factors inside one attribute do not separate: expression comes along with body pose. Four or more references degrade sharply. Clips skew to natural scenes, and effects that render text are scarce. Training stops at 480p. 720p and 4K are untested.
All 125 bench cases are single-video-reference. Multi-reference editing is the main claim, and it has no score on this bench. The three-reference result is a qualitative figure. The judge is Gemini 3.8 Flash, with no reported correlation to human raters. Commercial systems may not consume the same task-definition text, so part of the adherence gap can be an interface mismatch.
Synthesis stacks GPT Image 2, Wan2.2, Vista4D, and Qwen3-VL, and the same family filters the data. If the attribute is not fully removed from the source, the supervision is wrong. Two-reference supervision covers only two pairings, 50K each. About 7M of 10M examples are text-only, and the low RA on motion and VFX matches those smaller sets. The modest R²-RoPE drop is measured on a single-reference bench. It does not show that eight slots stay reliable at four references.