CAPEval: A Decoupled Caption Evaluation across Understanding and Generation
Zhipeng Liu, Haochen Wang, Zhaoxiang Zhang
cs.CV
2026-08-04
CAPEval scores captions by Coverage and Precision over 300 images and 50k facts: understanding tracks coverage, generation tracks precision, and the right captioner beats a bigger one.
Captions are the main supervision for vision-language models (VLMs) and text-to-image (T2I) generators, yet they are usually scored by a single scalar (BLEU, CIDEr, CLIPScore) that blends two different things: how much of the image's real information a caption covers, and how many of its claims the image actually supports. These pull in different directions, and it has been unclear which to optimize. CAPEval splits them apart to see whether coverage or precision drives downstream skill.
CAPEval uses human-written ground-truth captions and human-verified atomic checklists. There are 300 high-resolution images, each GT caption about 346 words, broken into about 50 atomic facts across 8 semantic dimensions, for 14,965 verified facts in total. Each candidate caption is generated by the captioner under test and judged by Qwen2.5-72B to compute:
A controlled end-to-end experiment then varies only the caption source. Ten captioners (InternVL3.5 1B/4B/8B/38B, Qwen3-VL 4B/8B/32B, LLaVA-OV-1.5 4B/8B, GLM-4.6V-Flash) each feed two VLM pipelines (CLIP+Vicuna, SigLIP+Qwen3) and two T2I pipelines (SD3.5M, Qwen-Image), with OLS regression measuring how C and P affect downstream performance.
Three findings:
This is a measurement result that directly guides data building: when training a VLM, push caption coverage; when training a T2I model, push precision; stop chasing a single score. The finding that a smaller but more precise captioner can beat a bigger one also matters for data cost. It is also a warning for evaluation itself: folding coverage and precision into one metric hides the dimension that is actually doing the work.
The authors list several: n is only 10 captioners, many p-values are marginal (0.026, 0.124), there are only 300 images, and everything depends on the Qwen2.5-72B judge, whose own biases can leak in. One more concern: the regression is linear OLS, so interactions or nonlinearity between coverage and precision (say, coverage turning harmful past some threshold) would not show up. The end-to-end experiment also fixes the downstream architectures, so whether the conclusions hold under other training recipes is unverified.