New 'Harmony' Metric Reveals How Average Scores Hide Uneven Model Performance
DanielKhashabi · x · 2026-08-21
Daniel Khashabi's team proposes "Harmony," a new metric to measure how uniformly a model's competence is distributed across different subdomains of a benchmark. The paper argues that relying on average scores is misleading, as overall accuracy can be disproportionately influenced by specific subdomains (e.g., Biological Concepts in ARC-Easy). They recommend reporting Harmony alongside accuracy to provide a more complete, distributionally reliable view of model performance.
More from Research
- Zetta ζ sets new SOTA on RoboCasa with 11.1x speedup — NielsRogge · 2026-08-21
- EMNLP paper: How VLMs map novel visual concepts to language vs humans — benno_krojer · 2026-08-21
- Andrew advocates for empirical and robust science of multiagent systems — soumitrashukla9 · 2026-08-21
- Stanford Study: LLMs Encode 'Current Year' Inconsistently, and Prompting Can't Fully Fix It — stanfordnlp · 2026-08-21
- Resource: One of the most rigorous math explanations of Transformers — stanfordnlp · 2026-08-21
- CfP: Learning and Reasoning with Graphs Workshop at BNAIC '26 — pbloemesquire · 2026-08-21