CPI-Bench: A Comprehensive,Practical and Intelligent Benchmark for Real-World Image Editing
Qinye Zhou, Jun Zheng, Yongchao Du, Yuan Wang, Zhengrui Chen, Zuan Gao, Taihang Hu, Chao Lin, Yefeng Shen, Xingjian Wang, Zhao Wang, Zhengtao Wu, Xiaoli Xu, Zhengze Xu, Hao Yan, Denghui Yang, Yuhang Yu, Huayu Zhang, Mingzhou Zhang, Mengting Chen
cs.CV
2026-08-15
Alibaba's CPI-Bench adds 3,778 image-editing samples across general, practical, and reasoning subsets; GPT-Image-2 scores 4.77 overall while the best open-source model reaches 3.37, and the resulting rankings align almost exactly with the human Arena leaderboard.
Image-editing models advanced fast; evaluation did not. Existing benchmarks (GEdit-Bench at 606 samples, ImgEdit at 811) stick to single-image tasks like recoloring a dress, and skip two things that matter for deployment: multi-image editing, such as swapping a hotpot base using a second reference photo, and reasoning-heavy instructions, such as drawing a labeled manufacturing diagram of a juicer. Scores have saturated as a result. On the older benchmarks, mainstream models cluster around 4.0 with between-model variance of 0.04-0.06, which separates nobody.
CPI-Bench has three escalating subsets. CPI-General holds 2,039 samples over 24 subtasks, including 10 multi-image tasks (reference-based outfit swap, cross-image style transfer, reference-viewpoint change) that prior benchmarks never tested. CPI-Practical covers 51 real applications with 558 samples across portrait retouching, e-commerce ads, interior design, and content creation, from ID-photo background swaps to marketing posters. CPI-Intelligent carries 1,181 reasoning instances over 8 expert domains and 67 sub-disciplines, curated from the authors' earlier ExpertVerse dataset. A VLM judge scores each output 1-5 on instruction adherence, visual naturalness, and physical/detail consistency, with task-specific prompt variants for text editing and identity-consistent tasks. The data pipeline runs taxonomy definition, image-pool curation, VLM-generated instructions with multi-round human filtering, and face anonymization via face swapping.
| Dimension | Top closed | Top open | Gap |
| CPI-General (single-image) | GPT-Image-2, 4.74 | FireRed, 4.56 | 0.18 |
| CPI-General (multi-image) | GPT-Image-2, 4.51 | Qwen-Edit-2511, 3.62 | 0.89 |
| CPI-Practical | GPT-Image-2, 4.69 | FireRed, 3.43 | 1.26 |
| CPI-Intelligent | GPT-Image-2, 4.77 | FireRed, 3.74 | 1.03 |
| CPI-Overall | GPT-Image-2, 4.77 | Qwen-Edit-2511, 3.37 | 1.40 |
Three structural facts. Single-image editing is saturated: everyone sits near 4.5. Multi-image editing is the open/closed gap: closed-source models clear 4.0 while open models plateau near 3.0. On CPI-Practical, open-source models collapse to 1.0-1.5 on Chinese-English text translation, the worst single category in the suite. Between-model variance rises from 0.04-0.06 on legacy benchmarks to 0.73 on CPI-Intelligent. The credibility check: CPI-Bench's model ranking matches the human Arena leaderboard almost position for position, with only FLUX.2-klein-9B off by one rank, and it achieves the highest Spearman correlation and lowest MAE among all benchmarks compared. GEdit, ImgEdit, and REDEdit rankings diverge visibly from Arena.
For editing-model teams this is a gap list: the open-source deficit concentrates in multi-image consistency and reasoning-based editing, not basic edits. For buyers, the Practical subset maps to deployment scenarios directly. The validation move is worth copying by any benchmark author; anchoring an automated benchmark against a human-preference leaderboard is the cheapest defense against measuring the wrong thing.
Alibaba built the benchmark and Seedream models rank second in its tables; the Arena alignment mitigates but does not eliminate the conflict-of-interest question, and the paper does not discuss it. The VLM judge's agreement with human raters is never reported directly; Arena rank alignment is indirect evidence. CPI-Intelligent reuses the authors' own ExpertVerse data. CPI-Practical spreads 558 samples over 51 tasks, roughly 11 per task, so per-category scores carry real noise.