MMTEB Leaderboard Suffers from Training Set Contamination
antoine_chaffin · x · 2026-07-15
This post highlights that ranking by SpartQA and MIRACL yields interesting yet "dangerous" results.
Replies clarify that these are official tasks on the MMTEB leaderboard. Currently, no model achieves naturally high scores on these tasks unless it was trained directly on these specific datasets. Essentially, some high scores reflect data leakage/memorization rather than true generalization; some "global #1" models might actually perform worse overall on other tasks.
Related event: MMTEB Leaderboard Faces Training Data Contamination Controversy(3 posts)→
More from Research
- DeepMind paper: LLMs can't sample randomly on their own, only convert given seeds — rohanpaul_ai · 2026-10-27
- DeepMind paper: LLMs can't sample randomly on their own — pass random numbers from code instead — rohanpaul_ai · 2026-10-27
- World Model Assigns Weight and Friction to Objects, Then Simulates with a Physics Engine — jonstephens85 · 2026-10-27
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- Mathathon 2.0 launches with Terence Tao's SAIR, $1.5M in compute for teams — _sathvikr · 2026-10-27
- The 'sweet lesson' is real: two kinds of ML problems need opposite strategies — mike64_t · 2026-10-27