MMLU vs GPQA: Key Differences and Which to Use

MMLU vs GPQA is breadth against depth. MMLU is a 2020 exam of about 14,000 four option questions across 57 subjects, gathered from practice tests and course material, so it mostly measures how much a model remembers. GPQA is a 2023 set of 448 graduate level biology, physics and chemistry questions written by PhD experts and checked so that skilled outsiders with web access still fail, so it measures expert reasoning. GPQA is far harder, and if you must read one of the two in 2026, read GPQA Diamond, but know that both are now close to their ceiling for frontier models.

This guide sets the two side by side, explains why the same model scores so differently on them, shows why a small test like GPQA Diamond needs careful reading, and gives a simple rule for which number to trust. For the basics of how benchmarks work, start with LLM benchmarks explained.

MMLU vs GPQA at a glance

In plain words: the two share a format (four choice questions) and almost nothing else. MMLU asks many easier questions about many topics. GPQA asks few, very hard questions about three sciences.

What MMLU measures

MMLU, short for Massive Multitask Language Understanding, was built by Dan Hendrycks and colleagues to test how much knowledge a model absorbs during pretraining (paper). The questions span 57 tasks and were collected from practice exams and course material that anyone could find. Its strength was coverage: one score summarized a model's grasp of history, law, medicine, math and more.

That design is also why it aged. Because the questions came from public sources and the test itself is everywhere online, models can meet them, or close copies, during training. Scores bunched up near the top, and researchers found the answer key itself was noisy. The MMLU-Redux team reannotated 5,700 questions and estimated that 6.49 percent of all MMLU questions contain errors (paper). When a test has that much label noise, a one or two point lead is not a real difference.

What GPQA measures

GPQA stands for Graduate Level Google Proof Q&A. David Rein and coauthors hired 61 contractors through Upwork, each with or working toward a PhD, to write questions in their own field that other experts could answer but outsiders could not (paper). Each question went through a strict pipeline:

1. An expert writes the question, four options and an explanation.
2. A second expert in the same field answers it and gives feedback, and the writer revises.
3. A third expert answers the revised question.
4. Three nonexperts (PhDs from other fields) try it with unrestricted web access but no AI assistants.

The nonexperts spent an average of 37 minutes per question and still reached only 34 percent accuracy, which is what "Google proof" means here. Experts in the same field reached 65 percent, or 74 percent once clear slips they admitted afterward were set aside. Writers were paid bonuses when experts got their question right and when nonexperts got it wrong, so the incentives pushed toward hard but answerable questions.

Main, Diamond and Extended

The paper collected 564 questions, held back 18 as a private set, and released three nested versions:

• Extended (546 questions): everything released.
• Main (448 questions): drops questions where both experts missed and all three nonexperts got it right. The authors recommend this set for experiments.
• Diamond (198 questions): keeps only questions where both experts answered correctly (or the second one clearly explained their own mistake) and most nonexperts failed.

Model launch posts almost always report Diamond, so when someone says "GPQA" in 2026 they usually mean those 198 questions. Physics is the largest domain in the extended set (227 questions), then chemistry (214) and biology (105).

Is GPQA harder than MMLU?

Yes, by a wide margin, and the launch numbers show it. GPT-3 scored 43.9 percent on MMLU when that paper came out, and GPT-4 scored 86.4 percent when it launched in 2023 (GPT-4 technical report). In the same year, the GPQA paper found GPT-4 with few shot chain of thought reaching just 39 percent on the main set and 38.8 percent on Diamond, barely above the 25 percent you get by guessing. Llama-2-70B-chat and GPT-3.5 sat between 28 and 31 percent.

Three design choices explain the gap:

• Who wrote the questions. MMLU recycles existing exam material. GPQA questions were written fresh by specialists and filtered against both experts and outsiders.
• Recall versus reasoning. Many MMLU items can be answered by remembering a fact. GPQA questions typically chain several steps of domain reasoning, and the authors asked writers to make them answerable even without the options shown.
• Exposure. MMLU questions are reproduced all over the web. GPQA ships behind a gated download with a canary string so training pipelines can filter it, and the authors ask people not to post examples in plain text.

So the scores are not comparable. A model at 90 on MMLU and 60 on GPQA Diamond is not "worse at GPQA" in any simple sense; the two numbers sit on different scales built for different purposes.

The small sample problem with GPQA Diamond

MMLU has about 14,000 test questions, so one question is worth roughly 0.007 points. GPQA Diamond has 198, so one question is worth about half a point. That has real consequences.

Take a model that truly gets 90 percent of Diamond questions right. The standard error of a single run is about 2.1 points (the square root of 0.9 times 0.1 divided by 198), so a 95 percent interval is roughly plus or minus 4 points. Two models 2 points apart on one Diamond run are statistically tied. The GPQA authors flag this themselves: they write that the 448 question main set needs large effect sizes, such as 50 to 60 percent, to detect a difference with 80 percent power.

What to do about it:

• Prefer results averaged over several runs. Many leaderboards now show "mean@8" or similar, meaning the average of eight attempts.
• Look for confidence intervals and treat overlapping ones as ties.
• Check that the answer options were shuffled between runs. Epoch AI notes that it reorders choices randomly from run to run to avoid exploitable patterns.

How much of GPQA Diamond is wrong?

Less than MMLU, probably, but not zero. In May 2025, Epoch AI's Greg Burnham looked at why top scores had clustered around 83 percent (Epoch AI). Across 14 strong models, the questions most of them missed concentrated in about 40 of the 198, and organic chemistry made up 70 percent of that hard group versus 36 percent of the full set. Reviewing six of the most consistently missed questions, he judged about 2.25 of them invalid, and concluded that at least 90 percent of the benchmark is likely valid.

In 2026 an independent project, GPQA Diamond Clean, went further. It audited the set, excluded nine questions it judged defective, and rescored 268 model configurations on the remaining 189 (gpqa.ai). It is not an official update from the GPQA authors. Its October 2026 table shows how close the top has come to the ceiling: the leading configuration scores 96.0 percent on the original 198 questions and 98.9 percent on the cleaned 189, and 260 of the 268 scores rose after cleaning. When the best result on a test sits within a few questions of perfect, and some of the remaining misses may be bad answer keys, the test can no longer rank frontier models.

Where each benchmark stands in 2026

Neither benchmark drives the main independent rankings anymore. The methodology page for the Artificial Analysis Intelligence Index lists version 4.3.2 with ten evaluations, including Humanity's Last Exam, CritPt and several agent tasks, and neither MMLU nor GPQA Diamond is among them (methodology). MMLU lives on there in a different form: the separate Multilingual Index is based on Global-MMLU-Lite across 16 languages.

For frontier science reasoning, attention has moved to Humanity's Last Exam, which Artificial Analysis grades as open answers rather than a pick from four options, and to research level physics sets like CritPt. For knowledge breadth, MMLU-Pro replaced MMLU with ten options and harder questions, though it is near saturation too; our MMLU vs MMLU Pro guide covers that pair. If you track coding models, see MMLU vs SWE-bench instead, because neither MMLU nor GPQA says much about writing code.

Which should you use?

Pick by the question you need answered.

• Choosing a frontier model for a product: use neither as your main signal. Both are close to the ceiling, and a multiple choice exam tells you little about your own task. Build a small task specific test set instead, as described in how to evaluate LLM model performance.
• Comparing small or open weight models: GPQA Diamond still separates them, because smaller models sit well below the ceiling. Use averaged runs.
• Checking a fine tune for knowledge loss: MMLU (or MMLU-Pro) is a decent broad smoke test, as long as you run the same configuration before and after. Its size makes small regressions visible.
• Testing science reasoning specifically: GPQA, ideally Diamond with several runs, plus a harder open answer test.
• Reading old papers and model cards: MMLU is the common yardstick for 2021 to 2024 results; GPQA Diamond for 2024 and 2025.

For a wider view of which public scoreboards to watch, see LLM leaderboards compared, and for scores across many benchmarks in one place, the Epoch AI Benchmarking Hub.

How to run MMLU and GPQA yourself

EleutherAI's lm-evaluation-harness covers both (GPQA task docs). MMLU runs as the mmlu task. GPQA comes in variants for each subset, such as gpqa_diamond_zeroshot, gpqa_diamond_n_shot and gpqa_diamond_cot_zeroshot, and the gpqa tag runs them all. The GPQA dataset is gated on Hugging Face, so you must accept its terms and log in with a token before the harness can download it.

Write down these settings with every score, because they move results more than people expect:

1. Subset. GPQA main, Diamond and extended are different tests.
2. Shots and reasoning. Zero shot, few shot and chain of thought give different numbers; the GPQA paper reports all of them.
3. Answer extraction. The GPQA paper pulled answers with a simple pattern match on phrases like "answer is". A formatting quirk can cost points.
4. Runs and shuffling. Report how many runs you averaged and whether options were reordered.
5. Harness version. The harness changelog lists recent fixes to GPQA answer preprocessing, so note the version you used.

FAQ

Is GPQA harder than MMLU?

Yes. In 2023 GPT-4 scored about 86 percent on MMLU but only 39 percent on GPQA, against a 25 percent guessing baseline. GPQA questions were written by PhD experts and filtered so that skilled outsiders with web access still answered only about a third correctly.

What does Google proof mean in GPQA?

It means highly skilled nonexperts could not find the answers online. In the GPQA study, PhDs from other fields spent an average of 37 minutes per question with unrestricted web access, excluding AI assistants, and reached only 34 percent accuracy.

Should I compare MMLU vs GPQA Diamond scores directly?

No. They are different tests with different difficulty, subjects and sizes. Compare GPQA Diamond scores only with other GPQA Diamond scores run under similar settings, and the same for MMLU.

Why are GPQA Diamond scores so noisy?

Because it has only 198 questions, each worth about half a point. A single run on a model near 90 percent has a 95 percent margin of roughly plus or minus 4 points, so small gaps between models are usually ties. Averaged runs and confidence intervals help.

Is GPQA saturated in 2026?

At the frontier, close to it. The independent GPQA Diamond Clean leaderboard shows a top configuration at 96.0 percent on the original Diamond set in October 2026, and the Artificial Analysis Intelligence Index no longer includes it. It still separates smaller and open weight models.

Is MMLU still useful?

Mainly for continuity, multilingual testing and quick regression checks. Top models are near its ceiling, and an estimated 6.49 percent of its questions contain errors, so small differences at the top mean little.