MMLU vs MMLU Pro comes down to difficulty and design. MMLU is the original 2020 multiple choice knowledge exam with four answer options across 57 subjects; MMLU-Pro is its 2024 successor with ten options, more reasoning heavy questions and a cleanup pass that removed easy and broken items. Use MMLU-Pro if you have to pick one, because it still separates mid sized and open models where MMLU no longer can, but treat both as saturated at the frontier and never compare a score on one with a score on the other.
This guide sets the two side by side, explains why the numbers drop when a model moves from MMLU to MMLU-Pro, shows where each benchmark still earns its place in 2026, and gives you a checklist for reading the scores you see in model cards. If you need the basics on what benchmarks are first, start with our LLM benchmarks explained guide.
MMLU vs MMLU Pro at a glance
| Item | MMLU | MMLU-Pro |
|---|---|---|
| Released | 2020 (ICLR 2021 paper) | June 2024 (NeurIPS 2024 paper) |
| Built by | Hendrycks et al., UC Berkeley and partners | TIGER-Lab, University of Waterloo and partners |
| Questions | 15,908 total; about 14,000 in the test set | 12,032 test questions plus a 70 question validation set |
| Subjects | 57 tasks | 14 broader disciplines |
| Answer options | 4 | Up to 10 (83 percent of questions have all ten) |
| Random guess score | 25 percent | About 10 percent |
| Best prompting | Direct answers often match or beat chain of thought | Chain of thought clearly helps |
| Status in 2026 | Saturated, known label errors | Near saturation at the frontier, still useful lower down |
In plain words: MMLU-Pro is smaller, harder and less forgiving of guessing, and it rewards models that reason through a problem rather than recall a fact.
What MMLU measures
MMLU, short for Massive Multitask Language Understanding, was introduced by Dan Hendrycks and colleagues to test how much knowledge a model picks up during pretraining (paper). Its questions were collected by students from freely available practice exams and course material, and range from elementary mathematics to professional law and medicine. The authors estimated expert level accuracy at roughly 89.8 percent, and when the paper came out the largest GPT-3 model scored 43.9 percent.
For several years MMLU was the headline number in nearly every model launch. That popularity is exactly what wore it out. Scores bunched up near the top, and the remaining gap turned out to be partly noise in the answer key rather than missing knowledge.
What changed in MMLU Pro
The MMLU-Pro authors started from three specific complaints about the original: too few distractors, too much pure recall, and too many wrong or unanswerable questions (paper). Each fix maps to one complaint.
Ten options instead of four
With four options, a model that can rule out two wrong answers has a coin flip chance at the rest. MMLU-Pro expands each question to up to ten options, using GPT-4-Turbo to draft six extra plausible distractors, which human reviewers then checked. That pushes the random guess baseline from 25 percent down to about 10 percent and makes shortcut elimination much less effective. After review, 83 percent of questions kept all ten options; the average is 9.47.
Harder, reasoning heavy questions
The team ran eight small open models over MMLU and threw out any question that more than four of them answered correctly. That filter removed 5,886 questions as too easy. They then added college level problems from a STEM problem website, TheoremQA and SciBench. About 57 percent of the final set still comes from MMLU, but subjects such as engineering, chemistry and physics are now drawn mostly from the new sources, many of which need multistep calculation. Law, history, philosophy and health remain entirely MMLU-derived, just filtered.
A cleanup pass, with limits
Every MMLU-Pro question went through two rounds of expert review, one to check answers and one, assisted by Gemini 1.5 Pro, to catch distractors that were actually correct. The review flagged 350 incorrect answers inherited from MMLU alone. The cleanup was not perfect: the maintainers still publish corrections on the dataset page, including 15 medical answers fixed in April 2025 after review by clinicians, and a January 2026 fix for stray leading spaces in some answer options that a model could have exploited as a formatting shortcut.
Why the scores are not comparable
The same model scores very differently on the two tests, and not by a fixed amount. In the MMLU-Pro paper, accuracy fell by 16 to 33 percentage points compared with MMLU. In the comparison table on the dataset card, GPT-4o dropped from 88.7 percent on MMLU to 72.6 percent on MMLU-Pro, while Claude 3 Sonnet fell from 81.5 to 55.1. A model that looks close to another on MMLU can be far apart on MMLU-Pro.
Three more differences make cross benchmark comparisons misleading:
- Spread. On MMLU, GPT-4o and GPT-4-Turbo were about one point apart. On MMLU-Pro the gap was about nine points. MMLU-Pro is simply better at telling models apart.
- Prompt sensitivity. Across 24 prompt styles, MMLU scores moved by 4 to 5 points, with peaks near 11. MMLU-Pro scores moved by about 2 points, with a maximum of 3.74. A single MMLU number therefore carries more hidden variance.
- Reasoning. On MMLU, chain of thought prompting barely helped and sometimes hurt; GPT-4-Turbo did 0.2 points worse with it. On MMLU-Pro, chain of thought lifted GPT-4o by 19 points. If one lab reports MMLU-Pro with reasoning and another without, the numbers are not measuring the same thing.
Is MMLU Pro saturated too?
Largely, yes, at the top of the market. When Artificial Analysis released version 4.0 of its Intelligence Index in January 2026, it removed MMLU-Pro, along with AIME 2025 and LiveCodeBench, and its published methodology lists the change in the version history (methodology). Coverage of that release noted that Gemini 3 Pro Preview had already reached 90.1 percent on MMLU-Pro (The Batch). Hugging Face retired its Open LLM Leaderboard, which had used MMLU-Pro as one of six core benchmarks, in March 2025.
MMLU-Pro is not dead, though. Open weight model cards still report it, and the self reported results shown on the dataset page in October 2026 cluster in the high 80s, so there is still some room between strong and middling models. For frontier comparisons, harder tests have taken over: GPQA Diamond for graduate level science and Humanity's Last Exam for questions at the edge of expert knowledge. Our LLM leaderboards compared guide covers which aggregators to watch.
Which should you use?
Pick by the question you are trying to answer, not by habit.
- Comparing frontier models for a product decision: use neither as your main signal. Both are close to the ceiling, and a multiple choice exam says little about how a model handles your task. Run task specific evaluations instead, as described in how to evaluate LLM applications.
- Comparing small or open weight models, or tracking your own fine tune: MMLU-Pro is the better choice. It spreads models out more, is less sensitive to prompt wording, and its ten options make lucky guessing rarer.
- Checking for regressions after fine tuning: either works as a broad knowledge smoke test, as long as you run the exact same configuration before and after.
- Comparing against older published results: MMLU, because that is what papers from 2021 to 2024 reported. If answer key errors worry you, also look at MMLU-Redux.
What is MMLU-Redux? It is a manually reannotated subset of 5,700 MMLU questions from researchers at the University of Edinburgh and partners (paper). They estimate that 6.49 percent of all MMLU questions contain errors, and found problems in 57 percent of the sampled Virology questions. Rescoring on the clean subset changed model rankings in several subjects, which is a strong argument against reading small MMLU differences as real.
How to run MMLU and MMLU Pro yourself
The simplest route is EleutherAI's lm-evaluation-harness, which ships ready made task groups for both benchmarks: mmlu and mmlu_pro, plus mmlu-redux for the cleaned subset (task list). If you want to reproduce the official MMLU-Pro numbers, the TIGER-Lab repository includes its own evaluation scripts and the cached predictions behind its leaderboard (GitHub). The original MMLU questions and code live in the authors' repository linked from the paper. For a full MMLU walkthrough with commands, see how to use the MMLU benchmark.
Whatever tool you use, write down the settings, because they move the score more than people expect:
- Shots. The MMLU-Pro paper used five shot chain of thought prompting for most models, drawing examples from the validation set. MMLU is commonly run five shot.
- Reasoning on or off. Report whether the model was allowed to think step by step, and at what reasoning setting.
- Answer extraction. The extraction described in the MMLU-Pro paper pulls the letter from text such as "the answer is (C)" and falls back to a random choice if it can't find one. A formatting quirk can therefore cost real points.
- Dataset version. MMLU-Pro has had several answer corrections since 2024, so note the date or commit of the copy you used.
For a wider look at harnesses, see our LLM evaluation frameworks comparison.
How to read a reported score
When a model card says "MMLU-Pro 85," ask four questions before you believe it means anything:
- Which benchmark exactly? MMLU, MMLU-Pro, MMLU-Redux and multilingual variants are different tests with different ceilings.
- Who ran it? Self reported numbers and independent runs often differ because of prompts, shots and extraction.
- Under what settings? Zero shot versus five shot and reasoning versus direct answers can swing results by many points on MMLU-Pro.
- How big is the gap? On a saturated test, a one or two point lead is within the noise created by label errors and prompt wording.
If a vendor only highlights MMLU in 2026, treat it as a sign that harder benchmarks were less flattering.
FAQ
Is MMLU Pro harder than MMLU?
Yes. In the original MMLU-Pro paper, models scored 16 to 33 percentage points lower than on MMLU. The ten answer options, the removal of easy questions and the added multistep STEM problems all push scores down.
Can you compare MMLU and MMLU Pro scores?
No. They are different tests with different random baselines (25 percent versus about 10 percent) and different question mixes, and the drop between them varies by model. Only compare MMLU-Pro scores with other MMLU-Pro scores run under similar settings.
What is a good MMLU Pro score?
It depends on the model class. GPT-4o set the top score of 72.6 percent when the benchmark launched in 2024; by early 2026 a frontier model had reached 90 percent, and the best self reported open weight results now sit in the high 80s. Judge a score against models of similar size, evaluated the same way.
Why does MMLU Pro use ten answer options?
To cut the payoff from guessing and from eliminating obviously wrong answers. More plausible distractors lower the chance baseline to about 10 percent and, according to the authors, make scores less sensitive to prompt wording.
Is MMLU still used in 2026?
Mostly for continuity and smaller models. Frontier labs and independent indexes have moved to harder evaluations because top models are near MMLU's ceiling and roughly 6.5 percent of its questions are estimated to contain errors.
Does MMLU Pro test reasoning or knowledge?
Both. Law, history, philosophy and health questions remain knowledge heavy, while math, physics, chemistry and engineering lean on multistep calculation. That mix is why chain of thought prompting helps far more on MMLU-Pro than on MMLU.
