VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks
Tengju Ru, Weitong Lian, Zecong Tang, Lingyi Meng, Haoran Li, Zhejun Cui, Yichen Zhu, Hangshuo Cao, Qi Kang, Yechi Liu, Kaixuan Wang, Yu-Jie Yuan, Chunwei Wang, Yu Zhang, Bo Dai
cs.CV
2026-11-03
VidUEU-Agent builds 100K instructions from 8,268 raw videos. Fine-tuning lifts InternVL3-8B MMDU from 53.1 to 61.1 and Bagel GEdit overall from 6.527 to 6.951.
Unified multimodal models have to understand images, edit them from instructions, and interleave both inside multi-turn dialogs. Those three kinds of supervision rarely come from one source. Static image-text pairs have no before and after. Hand-written edit instructions are expensive. Raw video carries temporal order, but it is redundant and noisy, and it is not a training set.
Automatic pipelines mostly stop at understanding QA. LLaVA-Video writes video questions from sampled frames and does not cover editing or unified tasks. The missing piece is a repeatable procedure: raw video in, three instruction types out, with the mix updated from downstream scores.
VidUEU-Agent runs four stages, then retargets the data mix from training feedback. DeepSeek-V3 is the controller. Qwen3.5-9B is the vision-language model.
PySceneDetect splits each video into shots. The middle frame of a shot becomes a shot summary. Consecutive shots in a continuous scene form a chapter, and chapters collapse into a video summary. Gate A drops shots that are uninformative or visually poor.
Remaining shots are sampled at a fixed interval. Gate B combines tool quality statistics with a VLM check, deletes bad frames, and captions the rest. Tools are assigned at chapter level and shot level, and each segment keeps only a small set. Tool outputs are rewritten in plain language and, together with the three summaries, turned into a usefulness score per frame. A duration-based budget is spread across shots. Flat shots can receive zero. High-scoring candidates then pass an embedding cosine-similarity check. Weighted facility-location greedy selection keeps frames that add coverage and suppresses near-duplicate neighbors.
Metadata keeps only signals that pass rule checks. Differences between adjacent keyframes are normalized into atomic edits such as add, remove, move, and switch on or off. Ego-motion and exposure flicker in driving footage do not count as object edits. Frames, captions, signals, transitions, and a clip summary are bound into one record.
Gate C scores information, coherence, and overall quality, drops unfit records, and tags what remains as understanding, editing, unified, or a combination. Every sample must cite the signal ids it used. Edit pairs are limited to one or two changes inside the same shot, and the instruction tells the model to leave everything else alone.
Each iteration builds 30K samples, jointly fine-tunes Bagel from the same initial checkpoint, and reads MMDU, GEdit, and WEAVE. The agent updates target ratios over capability, domain, and task type by sampling scarce video types more often, changing frame density, keyframe budget, and score weights, and downsampling tasks that overshoot. The 100K set scales the ratio from iteration 4.
The pool is 8,268 videos and about 8,505 semantic segments from OpenVid-1M, Panda-70M, Wikimedia, YouTube, FineGym, BDD100K, and CodeVidCL. Nearly 20K metadata records expand into 40K understanding, 30K editing, and 30K unified samples across five scenarios and nine task types. A typical unified item asks the model to produce a target state and then pick the candidate image that matches the change.
Fine-tuning is LoRA, so only the inserted low-rank weights move. InternVL3-8B sees only the understanding subset. The setup text says Bagel-7BMoT, while the tables list 14B. MoT means a mixture of transformers, with separate branches for understanding and generation. The three capability subsets are tuned separately, so the best cells below are not one checkpoint.
| Model | Benchmark | Before | After |
| InternVL3-8B | MMT-Bench | 0.546 | 0.596 |
| InternVL3-8B | MME | 2308.0 | 2369.3 |
| InternVL3-8B | LLaVA-Bench | 69.0 | 70.8 |
| InternVL3-8B | MMDU | 53.1 | 61.1 |
| Bagel | MMT-Bench | 0.645 | 0.651 |
| Bagel | MME | 1573.6 | 1900.2 |
| Bagel | LLaVA-Bench | 47.5 | 57.9 |
| Bagel | MMDU | 28.5 | 33.2 |
MMDU checks whether multi-image, multi-turn answers stay tied to the pictures. InternVL3-8B moves from 53.1 to 61.1 on MMDU and from 2308.0 to 2369.3 on MME, above Qwen2.5-VL-7B in the same table (59.5 and 2308.6). Qwen was not trained on VidUEU-100K, so that comparison is an external reference. LLaVA-Bench only moves from 69.0 to 70.8, well under Qwen2.5-VL-7B at 82.7. Bagel's MMT-Bench moves from 0.645 to 0.651, and that start was already the highest in the table. Its MME rises from 1573.6 to 1900.2 and still sits under InternVL3 before fine-tuning.
On GEdit-Bench-En, semantic consistency (GSC) goes from 7.358 to 7.565, perceptual quality (GPQ) from 6.831 to 7.328, and overall (GO) from 6.527 to 6.951. Overall beats Step1X-Edit at 6.701 and Gemini-2.0-flash at 6.315, and the other two metrics are higher as well. Perceptual quality moves more than semantic consistency.
WEAVEbench measures interleaved understanding and generation. Bagel's overall score goes from 0.446 to 0.501. Logic goes from 0.406 to 0.480, Game from 0.365 to 0.419, Creation from 0.475 to 0.501, and Science from 0.378 to 0.426. That beats Bagel-Zebra at 0.449 and trails OmniGen2 at 0.609. Open-ended creation gains the least.
The iteration study uses joint fine-tuning, 30K samples per round, same initialization.
| Round | MMDU | GEdit | WEAVE |
| 1 | 29.7 | 6.541 | 0.448 |
| 2 | 29.2 | 6.796 | 0.452 |
| 3 | 31.4 | 6.873 | 0.458 |
| 4 | 32.8 | 6.901 | 0.499 |
| Final 100K | 33.0 | 7.065 | 0.526 |
Round 2 drops MMDU to 29.2 while editing and unified scores keep rising. Round 4 improves all three. At 100K, MMDU is 33.0, WEAVE is 0.526, and GEdit is 7.065. That 7.065 comes from joint training, not from the 6.951 run on the editing subset alone.
Ablations train on about 10K samples per capability. The full models use the 100K-scale subsets and are trained separately, so scale is mixed into the gap. At matched scale, the no-agent variant is worst: fixed-interval frames and direct VLM question generation score 22.0 MMDU, 5.041 GEdit, and 0.393 WEAVE. Removing Gate B yields 24.7, 5.175, and 0.404. Removing Gate A still scores 32.4, 6.773, and 0.497.
One video yields understanding, editing, and unified instructions, and the edit pairs are local changes by construction. Four rounds of benchmark scores set the sampling ratio. A new video domain can rerun the loop without a new rulebook.
The architecture is unchanged. Gains are uneven. Multi-image consistency moves. Single-image dialog barely does. The edit overall score is the cell that clears both Step1X-Edit and Gemini-2.0-flash in this table. The unified overall score rises by about 0.055 over Bagel and remains well short of OmniGen2 at 0.609. Teams that only need single-image QA get little from these 100K samples.
Iteration watches the test splits of MMDU, GEdit, and WEAVE. The paper calls those scores development-guided. Once the mix is updated against those three boards, the main text cannot separate real gains from tuning to the leaderboard. MMBench, MMStar, and VSI-Bench stay out of the feedback loop. Their numbers are only in the appendix, and the main text does not report them.
Ablations use about 10K samples. The full models use about 100K. The mild drop claimed for removing Gate A compares 32.4 at 10K with 33.2 at 100K. A matched-scale full pipeline is not listed. Gates and tool evidence clearly point the right way. A per-module point value is not established.
Long interleaved dialogs still pass early mistakes forward. In the case studies, early turns follow instructions better after fine-tuning, and later turns inherit the errors.
DeepSeek-V3 and Qwen3.5-9B both write and check the data, and Gate C is also a VLM judgment. Requiring signal-id citations removes some empty samples. Dropping that rule scores 29.4 MMDU, 6.031 GEdit, and 0.470 WEAVE, still at about 10K samples. Edits are restricted to one or two visible changes, so large redraws are out of scope. Pass rates and call cost are unreported. Much of the source pool is already curated, Gate A barely matters, and nothing here shows the pipeline would survive unfiltered web video.