Hugging Face researcher admits the decision index now faces MTEB's leaderboard-gaming problem
antoine_chaffin · x · 2026-10-13
Hugging Face researcher antoinechaffin concedes that the decision index now suffers the same issues MTEB has faced for years: there's still some signal, but big jumps deserve skepticism. Despite a strict data-filtering pipeline against eval leakage and deliberately excluding training sets that could make them in-domain, problems remain:
- Comparing OOD and in-domain models on one leaderboard means a single score can't capture the nuances
- MTEB's "zero-shot" column is a partial fix, but it relies on honesty and most people don't care
- His reluctant conclusion: to stay competitive on leaderboards you have to get in-domain, while trying not to sacrifice actual model quality
He also credits multimodalart's private eval sets for at least catching the worst benchhackers.
Related event: HF Researcher Admits Decision Index Polluted by In-Domain Gaming(2 posts)→
More from Models
- Meta's Muse Wonder scores 21 on ExploitGym vs Claude models' 191-300 — scaling01 · 2026-10-13
- Researcher flags Google Astra's bloated explanations, floats a conciseness benchmark — OfirPress · 2026-10-13
- OpenAI's latest blow to mathematicians: 'They have annihilated my field' — ThereWas · 2026-10-13
- Google AI Overviews denied a person had died, renewing criticism of AIO for news — lilyraynyc · 2026-10-13
- User feeds Claude Opus GTA frame-by-frame screenshots since it can't watch videos — adonis_singh · 2026-10-13
- Gemini Deep Research on web is really slow today, user reports — gaganghotra_ · 2026-10-13