CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition
Lai Wei, Chengqi Li, Jiapeng Li, Ruina Hu, Yue Wang, Weiran Huang
cs.CV, cs.AI, cs.CL, cs.LG
2026-07-28
CLBench-V tests multimodal context learning across three levels: grounding, applying, learning. On 3,443 instances and six frontier models, the best overall score is only 0.2847.
Context learning is when a model extracts and applies task-specific facts, rules, or procedures from given material instead of relying on what it memorized during pretraining. Existing benchmarks for this, like LongBench and CL-Bench, are almost entirely textual. But the material models actually need to learn from is usually multimodal: scientific findings live in figures and tables, financial indicators in annual reports, spatial decisions in maps, scenes, or web pages.
CLBench-V fills two gaps. It extends evaluation from text to mixed image-and-text, and it stops benchmarks from reporting only that the model failed to use the context, without saying which step broke. It decomposes multimodal context learning into three cumulative levels so failures can be localized to a specific stage.
The three levels are cumulative; a failure at a lower level blocks the higher ones:
The benchmark has 3,443 instances across 14 datasets, spanning science, finance, long documents, spatial reasoning, and web QA. Construction uses two pipelines: integrating public benchmarks with rejection sampling (Qwen3.5-Plus as inspector discards shortcut-solvable items), and two newly built domain tasks: financial ROE DuPont decomposition (ROE = net margin times asset turnover times equity multiplier, 206 instances) and paper-conclusion inference (recent medical papers; the model sees the version truncated before Results and must predict which findings the paper supports, 581 instances, with a scoring rule that penalizes over-prediction).
Overall and per-level scores (max 1.0):
| Model | Overall | L0 grounding | L1 application | L2 learning |
| InternVL3.5-30B-A3B | 0.2847 | 0.3080 | 0.1313 | 0.3536 |
| Kimi-K2.6 | 0.1991 | 0.1755 | 0.2582 | 0.2186 |
| Qwen3.6-27B | 0.1958 | 0.1865 | 0.2878 | 0.1398 |
| GPT-5.4 | 0.1894 | 0.2003 | 0.2814 | 0.0694 |
| Qwen3.5-Plus | 0.1862 | 0.1507 | 0.2954 | 0.1964 |
| Doubao-Seed-2.0-Lite | 0.1850 | 0.1797 | 0.2241 | 0.1655 |
The best overall is 0.2847, and no single model wins every dimension. InternVL3.5 is strongest at grounding (L0) and learning (L2), the latter carried by paper-conclusion inference; Qwen3.5-Plus is strongest at application (L1). Two anomalies stand out: InternVL3.5's L1 is only 0.1313, with a flat zero on financial ROE because the run exceeded the model's maximum context length; GPT-5.4's L2 is 0.0694, the lowest of any model. Several datasets stay hard for everyone, including visual-fact Pix2Fact, web-evidence BrowseComp-V3, CL-Bench-Table (max 0.0938), and spatial CourtSI.
The authors also examine judge reliability and context complexity. Swapping the judge noticeably changes open-ended scores; input length alone does not explain failure, since what matters is whether an input crosses a model's usable context limit, not how long it is within that limit; and image count affects models in opposite directions, helping some and distracting others.
Multimodal context learning is a real, distinct capability, and current frontier models (including GPT-5.4) are far from saturated, scoring in the twenties overall. The three-level split is especially useful for anyone building RAG or multimodal agents: it tells you whether your model fails at finding evidence, at applying values, or at learning new rules, instead of handing back a single opaque failure rate.
The authors call the results preliminary, with more model runs ongoing. They concede: heterogeneous data sources with inconsistent evaluation protocols; some subsets are small and should be treated as diagnostic probes; open-ended tasks rely on LLM judges with model-specific bias; the financial task scores only the final ROE, not intermediate reasoning.
A few things are more worrying. Only six models are evaluated, mostly proprietary or Chinese-lab, with no human baseline, so the 0.2847 headline lacks a reference point. Qwen3.6-27B is both the main judge and an evaluated model, so self-judge bias is not fully removed. Subset sizes vary wildly (CourtSI 645, CL-Bench-Table 96), and the overall score is dominated by MIRBench (1,063), paper-conclusion (581), and CourtSI (645). Financial ROE and CL-Bench-Table sit near zero for every model, which may reflect format or evaluation issues rather than pure capability gaps, inflating the unsaturated narrative.