CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models
Mukhtiar Ali, Harsh Dubey, Sugam Mishra, Chulwoo Pack
cs.CV, cs.IR, cs.MM
2026-08-05
CLIP-CC-Bench pairs 200 ninety-second movie clips with expert paragraph references and ranks VLMs via five embedding judges doing coarse and fine matching then Borda voting; VideoLLaMA3 tops 17 models with only 0.67, leaving large headroom.
Evaluations of video-language models (VLMs) mostly focus on short clips and single-sentence metrics (MSR-VTT, MSVD) or multiple-choice QA (Video-MME). Whether a model can turn a minute-long clip into a coherent paragraph-length story is barely assessed. The metrics do not help either: BLEU, ROUGE, and CIDEr only measure surface n-gram overlap and miss paraphrase and narrative structure, while LLM-as-judge is a black box with unstable scores. CLIP-CC-Bench fills that gap.
For data, the authors select 200 ninety-second clips from five hours of film, each paired with an expert-written paragraph reference, deliberately stripped of proper nouns and cultural references so models must rely on pure visual understanding rather than memorized associations. The evaluation framework uses the five strongest embedding models on the MTEB leaderboard as judges (GTE-Qwen2-7B, KaLM-Gemma3-12B, Llama-Embed-Nemotron-8B, NV-Embed-v2, Qwen3-Embedding-8B) and scores at two granularities: a coarse paragraph-level cosine similarity and a fine sentence-level F1 from greedy best-match alignment, combined as the harmonic mean HM-CF so a model strong on only one side gets pulled down. Final ranks come from a Borda vote across the five judges, which cancels each judge's scale bias.
Across 17 mainstream VLMs, VideoLLaMA3 takes the full Borda score (80/80) and ranks first with mean 0.67, followed by mPLUG-Owl3 (75, 0.66); LLaVA-OneVision and ViLAMP tie for third (67, 0.64); general-purpose Qwen2.5-72B and 32B sit at sixth and seventh; last-place LongVA reaches only 0.48. Two patterns stand out. First, every model scores higher on coarse than fine granularity (VideoLLaMA3: 0.82 vs 0.76 on KaLM), meaning they capture the gist but miss fine detail. Second, the ceiling is only 0.67, so paragraph-level description of long video has large headroom. On reliability, the five judges agree at Spearman 0.96 to 0.99 at the system level, and after 1,000 bootstrap resamples the ranking is highly stable (mean Kendall tau 0.98, the top model preserved 100% of the time), so the scores are not an artifact of any single judge.
Long-video understanding is the next battleground for VLMs, but without a solid paragraph-level benchmark there was no way to compare models fairly. This benchmark tackles the old unstable-scoring problem with an ensemble of judges plus Borda aggregation and a bootstrap stability check, so its ranking is more trustworthy than any single judge or metric. For anyone choosing or building long-video models, it offers a comparable leaderboard and a reproducible script.
The authors acknowledge the bounds: only 200 clips, all narrative film with no instructional, surveillance, or user-generated content, English only, and a single reference per clip even though a scene can be described many valid ways. Methodologically, semantic matching does not model the temporal or causal order of events, so "right events, wrong order" goes unpenalized. The automated scores have not yet been validated against large-scale human ratings either (the authors list sample-based human meta-evaluation as future work), so external validity still needs to be established.