Are We Done with MMLU?
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, Pasquale Minervini
cs.CL, cs.AI
2024-06-06
Edinburgh annotators relabel 5,700 MMLU items; stratified error rate is 6.49%, Virology is 57% dirty, and ranks flip on clean questions. Auto-detection tops out at F2 44.75.
MMLU is Hendrycks et al.'s 2021 57-subject four-choice exam, scraped from textbooks, AP/GRE/CPA papers, and web quizzes. It is still the default knowledge score in LLM reports. The items were human-labeled, the procedure is barely documented, and scattered blog and YouTube posts have pointed at bad items without counting error types by subject.
A wrong label mixes "the model does not know" with "the key itself is wrong." One Virology item on preventing Ebola outbreaks lists "send EU and US army teams into West Africa" as the MMLU answer; the Human Virology textbook key is rebuild medical infrastructure and train staff. Matching the official label on that item tells you nothing about virology skill.
Fourteen domain experts, led from the University of Edinburgh, labeled each item with a decision tree. They did not overwrite the answer key. The tree asks, in order:
Five leaf errors: bad question clarity, bad options clarity, no correct answer, multiple correct answers, wrong ground truth. They drew 100 random items from each of 57 subjects, 5,700 in total, and released that set as MMLU-Redux. Where a source textbook or exam was recoverable, they checked against it. Annotators' suggested correct options are not written back into MMLU.
The full benchmark has 14,042 items. They estimate the global error rate by weighting each subject's sample rate by its original size. Three annotators double-labeled five dirty subjects; Cohen's Kappa ran from 0.64 to 0.96.
They then scored HELM-leaderboard models two ways: on each full original subject, and on the verified-clean MMLU-Redux items, and watched ranks move. A second track asks whether models can find the errors: zero-shot, few-shot, and chain-of-thought prompts, BM25 retrieval of five Wikipedia or MS MARCO passages, and a Llama-3-8B-Instruct fine-tune on LabelChaos, a 264k synthetic set corrupted with the same five error types.
Stratified estimate: 6.49% of MMLU is erroneous. Breakdown: bad question clarity 2.47%, multiple correct answers 1.54%, wrong ground truth 1.42%, no correct answer 0.62%, bad options clarity 0.44%. Dirty subjects and clean ones differ by an order of magnitude.
| Subject | Clean of 100 | Dominant error |
| Virology | 43 | wrong ground truth on 33 items |
| Logical Fallacies | 74 | unclear stems on 14 items |
| College Chemistry | 75 | wrong ground truth on 21 items |
| Professional Law | 82 | multiple correct answers on 11 items |
| Management and 8 other subjects | 100 | all 100 sampled items clean |
Virology's mass mislabels look like failed parses of the Human Virology question bank. College Chemistry cuts multi-line stems so option A is leftover question text and D disappears; some original E keys were dropped to force four choices. Formal Logic marks (F∧L)∧¬C correct and F∧L∧¬C incorrect, even though the formulas are equivalent. Professional Law assumes US jurisdiction without saying so. Almost every Global Facts item needed an external lookup; Our World in Data and Transparency International disagreed on a 2013 Ethiopia corruption item.
Ranks move. On Virology, Llama 3.1 Instruct Turbo 405B sits 16th on the HELM full-subject score (exact match 0.57) and 1st on verified-clean items (0.93). On Human Sexuality, GPT-4 (0613) falls from 5th (0.91) to last of the ten listed models (0.43); Gemini 1.5 Pro on the same subject jumps from 55th (0.37) to 6th (0.94). On Professional Law and Formal Logic, exact match on erroneous items is not lower than on clean ones. The paper reads that as pretraining memorization of the original keys.
Automatic detection does not hold. Claude 3 Opus few-shot CoT is the best prompt setup: recall 48.85, F2 40.29. RAG can push recall to 83.91, with F2 only 41.92, mostly from extra false positives. A Llama-3-8B fine-tuned on synthetic dirt reaches F2 44.75 and recall 56.58, still far from a cleaner you would trust.
Anyone still quoting MMLU as a selling point or an ablation axis should treat 6.49% as material. Small dirty subjects such as Virology and College Chemistry can rewrite a ranking on their own. A large subject such as Professional Law, 18% errors in the sample times 1,534 items, pulls the weighted global rate up.
MMLU-Redux is usable today as a filter (drop already-flagged items) or as a test set for dataset-error detectors. It is not a corrected MMLU. The concurrent MMLU-Pro paper filtered and extended the original set; residual original errors remain.
You cannot wash the rest automatically. Claude's 80%+ recall comes with collapsed precision; the 8B fine-tune only gets F2 to 44.75. Cleaning the remaining items still needs people.
The 5,700 items cover only part of 14,042; 8,342 are untouched. The 6.49% stratified figure assumes each subject's 100-item error rate represents that subject. Professional Law and Miscellaneous, the big slices, dominate the estimate.
The protocol still depends on judgment: whether a stem lacks context, whether a law item has more than one defensible answer. Kappa on Human Sexuality is only 0.64. The paper says annotator bias is not ruled out.
The two columns in Table 2 are not the same evaluation. Left is HELM full-subject scores; right is the clean MMLU-Redux slice. Prompts and item counts may differ, so do not subtract the exact-match numbers. Rank movement is the claim that stands.
Suggested correct options were never written back, so the "re-eval on clean items" drops bad items rather than repairing them. Higher accuracy on erroneous Professional Law and Formal Logic items is only circumstantial evidence of memorization; there is no membership-inference test.