What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks
Meera Desai, Sang T. Truong, Hanna Wallach, Alex Chouldechova, A. Feder Cooper, Jean Garcia-Gathright, Daniel E. Ho, Abigail Z. Jacobs, Sanmi Koyejo, Nicholas Pangakis, Angelina Wang
cs.CY
2026-09-08
56 benchmarks, 53 models: same-label safety concepts often correlate weakly; capability concepts barely split; BBQ-accuracy tracks reasoning over bias.
Scores labeled reasoning, refusal, or bias get treated as compact summaries of capability and safety. They feed research priorities and procurement. Whether a benchmark measures the concept on its label has mostly stayed a qualitative complaint.
Social science already tests this with convergent validity (same-concept instruments should move together) and discriminant validity (different-concept instruments should not). AI benchmarks break that setup. Concepts are underspecified, so a low correlation between two reasoning suites can mean a bad instrument or two legitimate definitions. Almost no suite has itself been validated, so there is no gold standard. This COLM 2026 paper, from Michigan, Stanford, Microsoft Research and others, reframes both tests as group comparisons: do rankings converge inside an assigned concept, and do they discriminate across concepts. They run 53 instruction-tuned models on 56 benchmarks and release item-level and benchmark-level scores.
Each benchmark gets an assigned concept. HELM suites inherit HELM's targeted-evaluation labels; the rest are grouped from the introducing paper's wording. SGBench's 'safety discrimination' and Aegis's 'content safety' both become safety detection. That yields 17 capability suites over 4 concepts (reasoning, knowledge, comprehension, summarization) and 39 safety suites over 7. Models span 31 families, open weights from 0.5B to 685B. Suites larger than 1,000 items are cut to 1,000. All runs are zero-shot at temperature 1, scored with each paper's original metric.
The headline analysis is Spearman correlation of model rankings, with cluster bootstrap over model families. At item level they fit 1PL IRT. For every pair of binary-scored suites they compare a shared-ability model against two suite-specific ability models on a 20% hold-out, average AUC over 10 splits, and call the gap ΔAUC. Near zero means one latent trait is enough; large positive means the pair wants separate traits.
Four saturated suites drop out (normalized top-versus-median gap below 0.05): DecodingTrust Stereotype, CIVICS, BoolQ, IMDB. Four free-response exact-match suites drop because zero-shot rankings disagree with HELM: WikiFact, Dyck, bAbI, Synthetic Reasoning Abstract. That leaves 48 for correlations and 37 for IRT. On the 10 overlapping HELM suites they keep, mean Spearman with HELM rankings is 0.74; on the four they drop, it is 0.25. For suites with a prior suspicion, a relabeling statistic subtracts mean absolute Spearman correlation with the current concept from correlation with a hypothesized one. Positive supports retagging (5,000 family-level cluster bootstraps).
Safety concepts often fail to converge. Pairwise Spearman correlations for refusal, safety detection, and bias have wide interquartile ranges and often sit near or below zero. Mean ΔAUC is 0.012 [0.011, 0.012] inside capability concepts and 0.0323 [0.0319, 0.0326] inside safety concepts. The paper does not read this as poorly built safety suites; many safety constructs may be multi-dimensional.
Capability concepts often fail to discriminate. ΔAUC between reasoning and knowledge is 0.011, sitting between within-reasoning 0.016 and within-knowledge 0.002. Summarization is the exception. Rankings on ethics, bias, privacy, and unsafe-behavior suites often track capability more than they track their own label. Ethics versus capability ΔAUC is 0.012, matching the within-capability figure. That lines up with prior safetywashing results.
Refusal and over-refusal were designed to pull opposite ways, and they do. Models that refuse more tend to over-refuse less. Cross-concept ΔAUC is 0.062, the largest of any pairing.
Shared design can outrun shared concept. Bias suites cluster by template (gender or race variants of the same benchmark), not by demographic target. A partial Mantel test on reasoning, comprehension, and refusal finds format a stronger predictor than concept when format is coded three ways (multiple choice / exact-match free response / LLM-judge free response): βformat = 0.275 (p<0.0001) versus βconcept = 0.138 (p=0.003). Collapse format to LLM-judge versus everything else and format dominates (β=0.526) while concept drops out (β=-0.058, p=0.998). SGBench-mcq, a multiple-choice refusal suite, clusters with other multiple-choice suites, not with LLM-judge refusal suites.
Some labels look wrong. BBQ-accuracy is sold as bias and affiliates more with reasoning (relabeling 0.15, 95% CI [0.07, 0.23], p<0.001). Items are ambiguous social scenes whose correct answer is 'not answerable'; a biased model and a model that reasons poorly both miss. DecodingTrust-Fair affiliates more with knowledge (0.14, [0.05, 0.24], p=0.002), and the correlation is negative. OR-Bench keeps over-refusal; relabeling against reasoning, knowledge, and comprehension runs from -0.62 to -0.52. BBQ-accuracy is often the only bias number in recent commercial model cards.
| Comparison | Metric | Value |
| Within capability vs within safety | mean ΔAUC | 0.012 vs 0.0323 |
| Within reasoning / within knowledge / between | mean ΔAUC | 0.016 / 0.002 / 0.011 |
| Format vs concept (3-way Mantel) | β | 0.275 vs 0.138 |
| BBQ-accuracy relabeled as reasoning | relabeling | 0.15, p<0.001 |
| HELM overlap, retained set | rank Spearman | 0.74 |
Treating a jump in reasoning and a drop in bias as independent progress is the move this paper warns against. Many capability suites may be reading out the same general ability. Safety labels that share a name often do not line up. Score format, especially LLM-as-judge, can structure correlations more than the concept on the paper.
Someone shipping a new benchmark can score the released 53-model set, run the relabeling test, and skip standing up 56 suites from scratch. A model card that only reports BBQ-accuracy has not finished a bias evaluation. This is a methods transplant plus a large empirical audit, not a new training recipe.
Intended concepts are often underspecified. Weak correlation can be a bad instrument or a construct that was never unidimensional. The paper treats low correlation as a flag, not a verdict.
IRT assumes approximate unidimensionality, which safety suites may violate. Correlation and ΔAUC disagree on some pairs: CALM and EntityMatching have ranking r=0.57 but ΔAUC=0.063, so item patterns look like two traits even as totals move together. Discriminant analyses do not residualize out general ability; the authors suggest residualizing on the first principal component of the correlation matrix.
The protocol is zero-shot at temperature 1, unlike HELM's few-shot temperature 0, which is why four exact-match generation suites were dropped. The set is English, single-turn, and automatically scorable. Collection cost about 1,050 H200 GPU-hours. Splitting models before versus after October 2024 (n=27 / 26) leaves the directional findings intact.