MMTEB Leaderboard Suffers from Training Set Contamination
antoine_chaffin · x · 2026-07-15
This post highlights that ranking by SpartQA and MIRACL yields interesting yet "dangerous" results.
Replies clarify that these are official tasks on the MMTEB leaderboard. Currently, no model achieves naturally high scores on these tasks unless it was trained directly on these specific datasets. Essentially, some high scores reflect data leakage/memorization rather than true generalization; some "global #1" models might actually perform worse overall on other tasks.
Related event: MMTEB Leaderboard Faces Training Data Contamination Controversy(3 posts)→
More from Research
- Project APE finds verifier reliability drops when papers contain multiple errors — soumitrashukla9 · 2026-07-22
- Project APE says verifier costs fell about 90x in a year as Chinese open models lead — soumitrashukla9 · 2026-07-22
- OpenAI-linked paper says capability RL can make models more reward-seeking — MariusHobbhahn · 2026-07-22
- Project APE builds its verifier benchmark from 100 AI-written papers with injected errors — soumitrashukla9 · 2026-07-22
- Paper proposes a CRED taxonomy and benchmark to measure research-error detectors — soumitrashukla9 · 2026-07-22
- OpenAI says long-horizon models need safety and alignment checks across full action sequences — rhiever · 2026-07-22