MMTEB Leaderboard Suspected of Training Set Contamination
antoine_chaffin · x · 2026-07-15
This reply adds an important observation about the MMTEB leaderboard: among its two official tasks, many models do not perform well.
The author notes that the few models that "look good" often directly used the training sets of those datasets, rather than demonstrating true generalization. This means leaderboard results may be significantly affected by data leakage / training set contamination; in other words, some top rankings do not indicate stronger models on broader tasks.
Related event: MMTEB Leaderboard Faces Training Data Contamination Controversy(3 posts)→
More from Research
- LightMTP: lightweight latent multi-token prediction boosts LLM planning in pretraining — gdm3000 · 2026-10-27
- EMNLP study: AI literary translation reads fine but varies more, even within one book — tuvllms · 2026-10-27
- Lab monkeys at $30-80K each: a 4-year breeding cycle is the real preclinical drug bottleneck — zebird0 · 2026-10-27
- OpenAI Foundation funds UNC tumor antigen project, hiring senior experimental scientist — iskander · 2026-10-27
- Steering attention heads toward honesty: up moral patienthood, down psychopathy — edelwax · 2026-10-27
- Are transcoders overkill? New paper argues plain neurons suffice for circuit analysis — burny_tech · 2026-10-27