Decision Index is facing MTEB's benchmark-gaming problem, its author says
antoine_chaffin · x · 2026-10-13
Antoine Chaffin, author of the Decision Index, says the leaderboard is now suffering the same fate as MTEB: contamination by in-domain-trained models.
- His team used strict data filtering to avoid eval leakage and deliberately excluded any training set in-domain with their target benchmarks, to measure generalization — which is why their model scored high even on tasks nobody knew about.
- But once others train in-domain, comparisons become unfair: beating an in-domain model with an out-of-domain one is very hard, and most people only look at the top score. That's why MTEB lost its value as a model-selection metric.
- Going forward they'll likely make their models in-domain too, carefully and transparently.
Related event: HF Researcher Admits Decision Index Polluted by In-Domain Gaming(2 posts)→
More from Models
- Meta's Muse Wonder scores 21 on ExploitGym vs Claude models' 191-300 — scaling01 · 2026-10-13
- Researcher flags Google Astra's bloated explanations, floats a conciseness benchmark — OfirPress · 2026-10-13
- OpenAI's latest blow to mathematicians: 'They have annihilated my field' — ThereWas · 2026-10-13
- Google AI Overviews denied a person had died, renewing criticism of AIO for news — lilyraynyc · 2026-10-13
- User feeds Claude Opus GTA frame-by-frame screenshots since it can't watch videos — adonis_singh · 2026-10-13
- Gemini Deep Research on web is really slow today, user reports — gaganghotra_ · 2026-10-13