Study Finds ~6.5% of MMLU Benchmark Questions Mislabelled

A paper (arXiv:2406.04127) manually re-annotated MMLU questions and found that about 6.5% have incorrect ground truth, with the virology subset reaching a 57% error rate, raising concerns about model evaluations based on the benchmark.

2026-09-19 ~ 2026-09-19 · 2 related posts

• MMLU-Redux paper finds ~6.5% of MMLU questions contain labeling errors — PMinervini · 2026-09-19

1 near-duplicate retellings: PMinervini