Benchmark Health Index: auditing 158 LLM benchmarks with SKYEVAL and Alibaba DAMO
jiqizhixin · x · 2026-11-17
Researchers from SKYEVAL and Alibaba's DAMO Academy propose the Benchmark Health Index (BHI) to audit LLM benchmarks, analyzing 158 benchmarks from reports of 117 models. BHI combines three signals — score discrimination, anti-saturation headroom, and capability-weighted adoption — via leave-one-benchmark-out calibration and CRITIC weighting. Robustness tests show removing 30% of models still leaves rankings with Spearman correlation above 0.97. The tool helps select and revisit evaluation sets as models improve.
More from Research
- NVIDIA releases USDCraft-10K robotics dataset with 3D, image and text modalities on Hugging Face — nvidia · 2026-11-17
- New design-build-test pipeline cuts protein plasmid costs ~24-fold — anshulkundaje · 2026-11-17
- BadEvolve: Poisoned Tasks Make Self-Evolving AI Agents Plant Their Own Backdoors at 79.6% Success — rohanpaul_ai · 2026-11-17
- Daniel Rock seminar: The Free Hypothesis — AI and Shifted Bottlenecks in Science — erikbryn · 2026-11-17
- Trajectory Labs postmortem: Gemini 4 Argon misalignment test agents reached real systems — sebkrier · 2026-11-17
- Researchers debate peer review: diversity of ideas vs broken proxy signals in academia — neurocy · 2026-11-17