Decision Index author admits benchmark suffers MTEB-style contamination

Antoine Chaffin admitted that the Decision Index leaderboard is repeating MTEB's old problem: in-domain training against the benchmark inflates scores. The leaderboard remains informative, but large jumps in scores should be viewed with skepticism.

2026-10-13 ~ 2026-10-13 · 3 related posts

• Decision Index is facing MTEB's benchmark-gaming problem, its author says — antoine_chaffin · 2026-10-13
• Hugging Face researcher admits the decision index now faces MTEB's leaderboard-gaming problem — antoine_chaffin · 2026-10-13
• MBZUAI researcher admits Decision Index now suffers the same benchmark overfitting as MTEB — ihor_step · 2026-10-13