MBZUAI researcher admits Decision Index now suffers the same benchmark overfitting as MTEB

ihor_step · x · 2026-10-13

MBZUAI researcher Antoine Chaffin admitted that the team's decision index now faces the same problem that has plagued MTEB for years — benchmark overfitting and potential data leakage inflating scores — cautioning readers to take major score bumps with a grain of salt. He noted the team already ran a strict data-filtering pipeline and deliberately excluded training sets that could put them in-domain. In response, user ihorstep, who tested the model on private real-world projects, called it among the best, and argued that overfitting to domain benchmarks reflects an industry-wide fixation on leaderboard rankings over solving real problems, which hurts innovation and more general models.

Original post →