JHU Paper Shows Aggregate Benchmark Scores Can Overstate Model Competence

DanielKhashabi · x · 2026-10-22

A Johns Hopkins paper (Uzunoglu, Li, Khashabi) studies whether aggregate benchmark scores faithfully reflect performance across subdomains.

Introduces Harmony, an entropy-based measure of how uniformly model performance spreads across subdomains, operationalizing benchmark-level distributional robustness.
Measured across 19 LLM benchmarks and five model families, robustness varies widely: low-Harmony benchmarks tend to let aggregate scores overstate broad competence.
Rebalancing benchmarks by pruning overrepresented subdomains substantially shifts aggregate scores for low-Harmony benchmarks; BoolQ stays stable while PubMedQA, a medically consequential domain, shows large, often statistically significant accuracy shifts.
Takeaway: aggregate scores can misrepresent model capability breadth, so reading leaderboards requires attention to subdomain distribution.

More from Models