Study claims harness configuration changes cause wild model ranking fluctuations
dair_ai · x · 2026-08-26
A study on harness evaluation examined how configurations affect outcomes. Twelve open-weight models answered 3,679 items from ARC, HellaSwag, MMLU, and TruthfulQA under 26 different configurations. Results showed gemma4-31b scoring anywhere from 31% to 89% based solely on the harness. The study found that 95.7% of the gap between models comes from config-fragile items, with four of the twelve models reaching rank one under some configuration. Benchmark compression methods tend to select for items most sensitive to configuration.
More from Research
- Schmidt Sciences awards $10M to 19 AI humanities research teams — OdedRechavi · 2026-10-15
- Mistral and CERN partner to build a Large Physics Model trained on billions of simulated collisions — MistralAI · 2026-10-15
- Global Transport coupling lifts guided flow generation: ImageNet FID 2.06 to 1.91 — mmbronstein · 2026-10-15
- Your breathing gives you away: Wi-Fi signals can reveal your breath rhythm — TinfoilTricorn · 2026-10-15
- Beyond text correlation: attaching a geometry simulator to ground model reasoning — gerardsans · 2026-10-15
- Sequen CEO Zoe Weil: Ranking Models Should Optimize Purchases, Not Clicks — TWIML AI Podcast · 2026-10-15