Open LLM Leaderboard v2: How to Use It in 2026

The Open LLM Leaderboard v2 was Hugging Face's second ranking of open language models. It launched on June 26, 2024 and scored every submitted model on six benchmarks (IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro), rescaled so that random guessing earns zero. Hugging Face retired it on March 13, 2025, so it no longer ranks new models. In 2026 it is still useful in two ways: its archived results are a free public dataset you can query, and its six benchmarks still run on your own hardware through the leaderboard task group in EleutherAI's evaluation harness.

So treat a v2 score as a fixed, reproducible snapshot of how a model did on 2024 era tests, never as a current ranking. This guide explains what the board measured, how its scores were calculated, what the archive actually contains (we loaded it and checked), how to rerun the tests, and where to compare open models now.

What the Open LLM Leaderboard v2 measured

Hugging Face picked six benchmarks covering knowledge, short and long reasoning, hard math and instruction following, and ran them all with the same code, prompts and settings (about page). Every run used EleutherAI's lm-evaluation-harness.

Benchmark	What it tests	Shots	Metric	Random guess baseline
IFEval	Following checkable format instructions ("use format y")	0	Strict accuracy, prompt and instruction level	0
BBH	24 hard BIG-Bench subtasks: logic, arithmetic, language	3	Normalized accuracy	Varies by subtask (2 to 19 options)
MATH Level 5	Hardest competition math problems, exact answer format	4	Exact match	0
GPQA	Graduate level biology, physics and chemistry questions	0	Normalized accuracy	25% (4 options)
MuSR	Murder mysteries, object placement, team allocation, about 1,000 words each	0	Normalized accuracy	50%, 20% or 33% by subtask
MMLU-Pro	Broad knowledge, 10 answer options instead of MMLU's 4	5	Accuracy	10%

The choice was deliberate. The launch post says Hugging Face wanted datasets with human review (MMLU-Pro and GPQA), wide use (BBH, IFEval, MATH), little contamination at the time (GPQA is gated; MuSR and MMLU-Pro were new) and fair metrics. It also says no benchmark that needs another model as a judge was included, partly because compute was tight (v2 launch post). If you want the background on two of these tests, see MMLU vs MMLU Pro and MMLU vs GPQA.

Why v2 replaced the first leaderboard

Version 1 had run for about a year on older tests such as ARC, HellaSwag, MMLU, TruthfulQA and GSM8K. The launch post lists three problems with them:

Saturation. Models were reaching human baseline performance on HellaSwag, MMLU and ARC, so the scores stopped separating models.
Contamination. Some instruction tuning sets contained GSM8K and TruthfulQA material, and models built on them scored well for the wrong reason.
Errors. MMLU had questions with wrong or missing answers, and GSM8K's setup used an end of generation token that unfairly cut off verbose models.

Version 2 also changed how the board ran. Hugging Face worked with EleutherAI to update the harness (adding chat templates and LoRA delta weights), hand checked the task implementations, added a "maintainer's highlight" list of notable models, and let signed in users vote on which submissions to evaluate first. The post also flagged a pattern that later shaped the retirement: models stacked from 20 or more rounds of fine tuning and merging sometimes did well on the leaderboard's own benchmarks without matching real use.

How the normalized score works

The headline number on v2 was an average of six normalized scores, not raw accuracy. Each raw score was rescaled so the random guess baseline becomes 0 and a perfect score becomes 100, and anything below chance was set to 0 (normalization docs):

def normalize_within_range(value, lower_bound, higher_bound):
    return (value - lower_bound) / (higher_bound - lower_bound)

# GPQA has 4 options, so chance is 0.25
raw = 0.60
print(normalize_within_range(raw, 0.25, 1.0) * 100)  # 46.67

For tests with subtasks (BBH and MuSR), each subtask is normalized against its own chance level and the results are averaged. MATH and IFEval are generative, so their baseline is already 0 and nothing changes.

This matters more than it looks, because it reweights the average. Hugging Face's own example: model A scores 0 on a hard generative test and 52 on a two option test; model B scores 10 and 40. The raw averages are 26 and 25, so A looks better. But 52 on a two option test is barely above a coin flip, and 40 is below it. After normalization both are near 0 on that test, B keeps its 10 point lead on the generative one, and B wins about 5 to 0. Any time you compare a v2 number with a raw accuracy from a paper or model card, check which kind it is. A GPQA "score" of 46.67 on v2 means 60% raw.

Why Hugging Face retired it

On March 13, 2025, Hugging Face announced the end of the leaderboard in a pinned post. It said the team had evaluated more than 13,000 models over two years, that benchmarks need to change as models move toward reasoning and assistant use, and that the board "could encourage people to hill climb irrelevant directions in the field" (retirement post). The same post pointed to more than 200 community leaderboards on the Hub. A follow up reply from the team said there is no single drop in replacement and linked a search tool, the Find a leaderboard Space, for leaderboards by domain (math, code, languages) or by category such as performance for edge devices.

That reasoning lines up with the general advice in our LLM benchmarks explained guide: a fixed public test gets easier to game the longer it stays the target.

What the archived v2 results show

The final table lives on in the open-llm-leaderboard/contents dataset, last updated March 20, 2025 (dataset). We downloaded it on October 30, 2026 and checked it with pandas. It has 4,576 rows (one per evaluated model and precision) and 36 columns, including both raw and normalized scores for each benchmark, parameter count, license, model type, a merge flag, an official provider flag and an estimated CO2 cost.

What stands out:

Community models dominate the top. The model type labels split into 1,785 rows fine tuned on specific datasets, 1,724 merges, 718 chat models, 275 pretrained models and 58 continuously pretrained ones. Nine of the top ten rows by average were uploaded by community accounts rather than the lab that trained the base model, and all ten are 32B to 78B models. The highest is MaziyarPanahi/calme-3.2-instruct-78b at 52.08. The best entry flagged as an official provider release is Qwen/Qwen2.5-72B-Instruct at 47.98, sixth overall.
GPQA barely moved off chance. The mean raw GPQA accuracy across all rows is 30.0%, against a 25% guessing baseline, which normalizes to a mean of 6.72. The best normalized GPQA score in the whole archive is 29.42 (47.1% raw), and 241 rows score exactly 0.
MuSR was nearly as hard. 1,125 rows score below 5 on normalized MuSR.
Only 470 rows are official provider releases. Filter on that column first if you want a clean view of lab releases rather than community variants.

Those numbers support Hugging Face's own reason for stopping: by early 2025 the top of the table was mostly community variants of large base models, often separated by a point or two, on a test mix where many models sat close to chance on two of the six benchmarks.

To explore it yourself:

import pandas as pd

url = "hf://datasets/open-llm-leaderboard/contents/data/train-00000-of-00001.parquet"
df = pd.read_parquet(url)  # needs pandas, pyarrow and huggingface_hub
official = df[df["Official Providers"]].sort_values("Average ⬆️", ascending=False)
print(official[["fullname", "#Params (B)", "Average ⬆️", "GPQA Raw", "MMLU-PRO Raw"]].head(10))

How to rerun the v2 benchmarks locally

The six benchmarks live on as the leaderboard group in EleutherAI's harness, described there as "an unchanging through time version" of the leaderboard evaluations (task README). This is the closest thing to a local copy of the board.

Install. pip install "lm-eval[hf,math,ifeval,sentencepiece]" plus PyTorch for your hardware. The math extra handles MATH answer checking with sympy.
Get GPQA access. The dataset is gated on Hugging Face, so accept its terms and log in with a token first, or the group will fail on that task.
Run. lm-eval --model hf --model_args pretrained=<model>,dtype=bfloat16 --tasks leaderboard --batch_size auto --output_path results/. For instruction models, Hugging Face says to add --apply_chat_template and --fewshot_as_multiturn, matching how the board ran them.
Start small. Run a single subgroup such as leaderboard_ifeval or leaderboard_mmlu_pro with --limit 20 to check the setup before a full run.
Normalize. The harness reports raw metrics. Apply the normalization above before comparing with the archive's normalized columns, or compare against the archive's "Raw" columns instead.

We tried this on October 30, 2026 with lm-eval 0.4.13, running leaderboard_ifeval with --apply_chat_template and --limit 10 on the tiny SmolLM2-135M-Instruct model on a CPU. It took about two and a half minutes and scored 0.20 prompt level strict accuracy, with a standard error of 0.13, which shows why a 10 prompt check proves the setup works and says nothing about the model. One catch: the first attempt generated every answer and then stopped at scoring, because the IFEval checker needs NLTK's punkt_tab data. Run python -c "import nltk; nltk.download('punkt_tab')" before you start.

Expect small differences from the archived numbers. Hugging Face notes that batch size changes results slightly because of padding, and the leaderboard originally ran on Hugging Face's own fork of the harness while some fixes were being merged upstream. Pin the harness version, record it next to your results, and only compare runs made the same way. Our guide on how to use the MMLU benchmark shows how much implementation choices can move a score. If you prefer Hugging Face's own tooling, Lighteval is its evaluation library, and OpenCompass and HELM publish their own suites.

Where to compare open models now

There is no single successor, so pick by what you need:

Benchmark leaderboards on the Hub. Hugging Face now lets dataset repos register as benchmarks (MMLU-Pro, GPQA, HLE and GSM8K are examples) and show a leaderboard built from scores stored in model repos as YAML files in a .eval_results/ folder. Anyone can add a score by pull request, which shows as "community" until merged, and scores run on Hugging Face Jobs with Inspect AI can carry a "verified" badge. Hugging Face still labels this a work in progress (Hub docs). Look for the verified badge before trusting a number.
Domain leaderboards. Use the Find a leaderboard Space for math, code, languages or edge device boards.
Independent general rankings. For a broad view that includes open weight models, see our LLM leaderboards compared guide, the Artificial Analysis Intelligence Index v5 explainer, LiveBench vs Artificial Analysis, and the Fireworks Specialized Intelligence Index for work specific boards.

How to read an old v2 score today

You will still see v2 numbers in model cards, papers and older blog posts. Before using one:

Check normalized or raw. A v2 average or GPQA figure is normalized unless it says "raw".
Check the date. Anything evaluated after March 2025 did not come from the official board.
Check model type. A merge or heavy fine tune near the top may be tuned toward these six tests.
Check the benchmark mix. v2 has no coding agent, tool use, long context retrieval or multimodal tests, so it says little about those skills.
Rerun if it matters. If you are choosing between two open models for real work, run the leaderboard group on both with the same settings, then add a test set from your own task, as our guide to evaluating LLM model performance explains.

FAQ

Is the Open LLM Leaderboard still available?

The results are. Hugging Face stopped evaluating models on March 13, 2025, but the full v2 results table remains public as the open-llm-leaderboard/contents dataset (4,576 rows when we checked on October 30, 2026), and the per model detail files stay on the Hub. You cannot submit a new model for an official score.

Is "Open LLM Leaderboard 2" the same as v2?

Yes. Both names refer to the version Hugging Face launched on June 26, 2024 with six new benchmarks and normalized scoring. The first version used older tests such as ARC, HellaSwag, MMLU, TruthfulQA and GSM8K, and its results were moved to a separate archive when v2 launched.

Why are GPQA scores on the leaderboard so low?

Because v2 subtracts the 25% chance of guessing on a four option question. In the archive the average raw GPQA accuracy is 30%, which becomes about 7 after normalization. Many 2024 open models were close to guessing on these graduate level science questions.

Can I get an official Open LLM Leaderboard score for my model today?

No official score exists any more, but you can produce a comparable one. Run the leaderboard task group in lm-evaluation-harness with the settings Hugging Face documented, normalize the results, and report the harness version. On the Hub, you can also add scores to a model repo through the newer evaluation results feature.

What replaced the Open LLM Leaderboard?

Nothing replaced it one for one. Hugging Face pointed users to more than 200 community leaderboards, its Find a leaderboard search Space, and later to benchmark leaderboards built from scores stored in model repos. For general rankings, independent boards such as Artificial Analysis and LiveBench cover open and closed models together.

Facts, prices and features in this guide were checked against official sources at the time of writing and can change. Always confirm details on the vendor's website.