2026-09-04
Eighteen demand scales annotate 16k items and profile 15 LLMs. No benchmark is both sensitive and specific. A demand forest hits AUROC 0.84 in-distribution and 0.75 OOD.
A 79.8% score for DeepSeek-R1 on a math contest set still does not tell you whether the next item from that set will succeed, let alone how far the number will fall on another suite also labeled reasoning. Aggregate accuracy is a property of a particular bundle of items, not a capability you can carry to a new task. Factor analysis and item-response theory extract latents, but those parameters live in the population of models and benchmarks used to fit them. Swap the population and the numbers age out.
Evaluation is missing two things: an account of why a model fails a given item, and a forecast for unseen items and unseen tasks.
The authors define 18 demand scales, discrete from 0 to 5+, written so a person can apply them and then applied at scale by GPT-4o. Eleven are elemental (attention and scan, logical and quantitative reasoning, mind modelling, and so on), five are knowledge domains such as natural and formal sciences, and three are extras: atypicality, volume, and unguessability. Atypicality tracks contamination. Volume tracks collage items that glue many easy subproblems together. Unguessability tracks multiple-choice funnels.
The scales are built as ratio scales with a true zero. Knowledge levels roughly track school stages from elementary to graduate. Applied to 20 benchmarks, 63 tasks and 16,108 items they yield 289,944 annotations, the ADeLe battery. Five human raters reach rWG between 0.70 and 0.91 (mean 0.83). Delphi consensus versus GPT-4o averages 0.86.
For each model and dimension they keep only items whose other demands do not exceed the level under study, then fit a logistic success curve. An ability of l means about 50% success at demand l. For prediction, a 19-dimensional demand vector feeds a random forest, against a GloVe forest and a fine-tuned LLaMA.
What a benchmark claims to measure and what it is sensitive to rarely match. Sensitivity needs mean ≥2 and s.d. ≥1 on the claimed dimension; specificity needs mean <2 on the rest. None of the 20 benchmarks meet both. SAT is 98.3% for GPT-4o and is sensitive to low atypicality. MedCalcBench is 88.0% and looks more like an attention-and-scan test than a medical-calculation test.
Under one reasoning label, DeepSeek-R1-Distilled-Qwen-14B sits near 4.5, 4.3 and 4.2 on quantitative, logical and inductive ability. Its scores are 90.50% on GSM8K, 61.80% on OlymMATH Easy, 59.10% on GPQA and 13.30% on OlymMATH Hard. OlymMATH Hard really does raise reasoning demand. GPQA does not; it loads extra knowledge dimensions.
Knowledge abilities track model size. Quantitative and logical reasoning, relevant-information spotting and mind modelling rise more in o1 and DeepSeek-R1 distillations. The two largest rungs of LLaMA and distilled Qwen already flatten.
| Setting | Demand-forest AUROC / ECE | Controls |
| In-distribution (10-fold) | weighted 0.84 / 0.01 | GloVe ECE 0.03, fine-tuned LLaMA ECE 0.04 |
| Best model (GPT-4o) | 0.882 | most predictable of the 15 |
| Task held out | 0.81 / 0.02 | LLaMA 0.79, GloVe 0.74 |
| Whole benchmark held out | 0.75 / 0.04 | both black-box controls drop harder |
The demand forest trains in about 4 seconds. The LLaMA assessor in the comparison costs 300 V100-hours. Subject accuracy on ADeLe runs from 0.102 (Babbage-002) to 0.843 (o1).
You can now say an item is a 4 on quantitative demand and a 1 on knowledge, instead of "it came from a math benchmark." Routing, refusal and regulatory mapping all need a ruler that transfers. Leaderboards collapse sensitivity and specificity into one percentage. This work splits them. Demand labels are not tied to a model population, so old items stay usable when the leaderboard turns over.
The catalogue does not cover navigation, multimodality or robotics. Level 5+ items are scarce and the scale will need a backward-compatible extension. Annotation and grading both use LLMs; open-ended agent tasks may not grade this cleanly. Ability curves are slices of these 20 benchmarks, not of all tasks. Benchmark-out-of-distribution AUROC of 0.75 is usable and is not a guarantee on a brand-new domain. High human-LLM agreement does not make the levels unbiased across cultures and specialties.