New Benchmark for Multi-Reference Audio-Visual Generation
KlingTeam · hf · 2026-07-17
This work proposes MultiRef-Compass to evaluate Multi-Reference Audio-Visual Generation (MR2AV).
Motivation
Existing benchmarks mostly focus on:
- Text-driven generation
- Single-reference subject preservation
- Consistency in only a part of image or audio/video
However, MR2AV requires models to simultaneously handle multiple references and generate synchronized audio-visual content, making it much harder.
Benchmark Design
The authors built a unified benchmark of 350 samples, covering:
- Multi-view subject preservation
- Multi-entity binding
- Person-object-scene composition
Evaluation Protocol
Defined 4 evaluation dimensions with 14 sub-metrics:
- Basic quality
- Reference consistency
- Audio-visual consistency
- Instruction following
Adopts an "automatic metrics + rejudging enhanced MLLM-as-a-Judge" framework, making the evaluation more scalable and auditable.
Experimental Conclusions
Across 8 representative MR2AV systems, the authors found current methods still have significant room for improvement across multiple dimensions, indicating this direction is far from solved.
More from Multimodal
- TischEins, a large Midjourney SREF library, relaunches with new design after security outage — tisch_eins · 2026-10-26
- MOSS-Image-SFT 4-step distillation tops 16.5K downloads on Hugging Face — _akhaliq · 2026-10-26
- Claude Code can now edit videos: a 10-prompt workflow from raw footage to final cut — TawohAwa · 2026-10-26
- One-sentence 3D worlds turn cinematic with CineMap and TapNow — gizakdag · 2026-10-26
- No Camera Needed: Creator Builds an Entire World in CineMap and Directs It Into Cinema — gizakdag · 2026-10-26
- Kling vs. Seedance: the Chinese AI video model rivalry — DavidmComfort · 2026-10-26