AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities
Yuqing Wen, Yukai Huang, Qianqian Xie, Jiangtao Wu, Yibin Lin, Yikai Gu, Jialu Chen, Yuanxing Zhang, Jiaheng Liu
cs.MM, cs.CV, cs.SD
2026-07-17
On 196 audio-video coupled edits, SOTA models fail cross-modal tasks and inflate fidelity scores by not editing. AVE-Agent hits 59.8 editing intent vs 42.4 for the best baseline.
Editing a real video means editing two coupled signals at once. Change a visible action and its sound has to change with it; remove an object on screen and its audio trace has to go too; rewrite a line of dialogue and the lip motion has to match while the background ambience stays put. Existing benchmarks test visual edits on silent clips or audio edits in isolation, so nobody has been measuring the case that actually matters: edits where audio and video have to move together. AVE-Compass fills that gap and, in the process, exposes a problem most evaluators had missed. A lot of models earn high scores by simply not editing.
The benchmark has 145 source videos, 196 audio-video coupled editing instructions, 2,688 fine-grained checklist items, and 28 fine-grained edit subcategories. It scores four dimensions: Instruction Following (IF, did the requested change land), Fidelity Preserving (FP, did the untouched content stay untouched), Realism (REAL, does it look and sound natural), and Editing Intent (EI = IF × FP, which demands both a correct edit and preserved context). Scoring uses an MLLM working through a checklist of atomic yes/no questions, with a modality-split design that makes separate calls for audio-only, video-only, and general questions to reduce cross-modal bias. A separate rubric scores realism, and seven automated metrics cover lip sync, AV sync, video aesthetics, subject consistency, motion smoothness, audio aesthetics, and speech quality.
The most useful design choice is the edit-response layer. High fidelity scores are often fake. A model that returns the input unchanged will obviously preserve everything, so its FP is high. The benchmark therefore computes a response rate (was the target modality actually modified) and recomputes FP and realism only over cases that were genuinely edited (gated FP). That cleanly separates "did not edit" from "edited badly."
AVE-Agent is the companion method, three agents in sequence. The Planner builds a scene profile, decomposes the instruction into a dependency-aware subtask DAG, and surfaces cross-modal consequences the instruction never stated. The Executor routes subtasks to video, audio, and speech branches, self-checks each one, and rewrites guidance on evaluator feedback, keeping the highest-scoring candidate. A Mixed Evaluator inspects the assembled clip and applies only the minimum correction (remix, regenerate one subtask, or replan), with a global keep-the-best rule to prevent regressions.
Main scores (MLLM judge, out of 100):
| Model | Editing Intent | Instruction Following | Fidelity Preserving | Realism |
| AVE-Agent (Wan) | 59.8 | 77.3 | 77.6 | 62.1 |
| Wan2.7 | 42.4 | 69.3 | 64.2 | 60.4 |
| HappyHorse | 41.3 | 66.9 | 63.9 | 63.0 |
| Gemini-Omni | 38.0 | 44.9 | 84.9 | 66.9 |
| Seedance | 26.6 | 37.4 | 81.7 | 69.2 |
| LTX2 | 15.2 | 70.1 | 30.6 | 64.1 |
AVE-Agent beats the best baseline by 17.4 points on editing intent and is the only system that lifts instruction following and fidelity together. The response-rate column is more telling. AVE-Agent modifies video 92.7% of the time and audio 94.4%, while Gemini-Omni modifies audio only 22.0% of the time and Seedance modifies video only 69.1%. Their high fidelity scores come from not editing. LTX2 fails the opposite way: it follows instructions but essentially regenerates the input video, hence a fidelity of 30.6. Removing retry refinement drops EI by 8.75 and removing prompt enhancement drops it by 7.76. Automated metrics correlate only weakly with the MLLM scores, and human-LLM agreement sits around 88% to 93%.
For anyone building or evaluating audio-video editing, the paper delivers a clear status check: cross-modal editing is not solved, and looking at video or audio in isolation will mislead you. The thing to copy immediately is the response-rate plus gated-fidelity measurement. Any pipeline that evaluates video or audio generation should first verify the model actually changed the target modality, or the fidelity number is meaningless. The AVE-Agent recipe of decomposition plus self-checking retry is a cheap, modular add-on to existing tools that lifts cross-modal editing quality without retraining anything.
The authors own the scale trade-off. 196 instructions is what the evaluation budget allows, since each clip needs closed-source editing APIs plus MLLM checklist judging, both billed per call. Using Seedance's official pricing, a single full pass for one model runs about $220 in video generation alone (196 instructions times roughly 7.4 seconds times about $0.15 per second), plus $10 to $15 in judging, roughly $230 to $235 total before failed generations, retries, storage, and human verification. A substantially larger benchmark would not be runnable as routine evaluation. AVE-Agent also depends on third-party tools whose behavior can drift, hurting long-term reproducibility. One gap the paper does not close: gated FP relies on the MLLM judging whether a modality was modified, and that judgment itself is only 88% to 93% reliable, so its own error could leak back into the gated score. The paper does not isolate that effect.