ARC-AGI vs Humanity's Last Exam is a contrast between learning and knowing. ARC-AGI gives a model a few examples of a puzzle it has never seen and checks whether it can work out the rule, with no special knowledge needed, while Humanity's Last Exam (HLE) asks closed ended questions written by subject experts that need deep academic knowledge plus reasoning. Use HLE Diamond to judge expert question answering, ARC-AGI-3 to judge how well an agent adapts to something new, and treat ARC-AGI-2 as a test the top models have nearly finished.

Launch posts often quote both side by side, as if they were two scores for one ability. They are not. Below: what each measures, how it is graded, what the scores say for the same models, and where each misleads. Every figure was read from the official boards and repositories on November 12, 2026.

ARC-AGI vs Humanity's Last Exam at a glance

ItemARC-AGI (ARC Prize Foundation)Humanity's Last Exam (CAIS and Scale AI)
What it measuresSkill acquisition on novel tasks (fluid intelligence)Expert academic knowledge and reasoning
Current versionsARC-AGI-1 (2019), ARC-AGI-2 (static grids), ARC-AGI-3 (interactive games)HLE (2,500 questions), HLE Diamond (1,000 questions, Sept 22, 2026)
Task formatColored grids, 1x1 to 30x30, or game environmentsText and image questions, multiple choice or short answer
Knowledge neededOnly "core knowledge priors" such as objects and countingGraduate and research level subject knowledge
GradingExact match on every cell, 2 attempts per test inputLLM judge compares the answer to the reference
Extra metricCost per task or total run costCalibration error from the model's stated confidence
ToolsNone by default (no code, no web)Closed book main score; separate web plus code setting
Human baselineEvery ARC-AGI-2 eval task solved by 2 or more people in 2 triesQuestions written by experts; no overall human score on the pages we read
Hidden test dataSemi private and private eval setsPrivate held out questions; gated public dataset

What ARC-AGI measures

ARC-AGI comes from François Chollet's 2019 paper "On the Measure of Intelligence", which defines intelligence as skill acquisition efficiency on unknown tasks. The ARC Prize explainer says it targets fluid intelligence rather than crystallized knowledge and uses only core knowledge priors that humans have from early childhood: no language, culture or trivia.

A static ARC task shows two to six input and output grid pairs and asks for the output of one to three new inputs. The test taker has to pick the output size as well as every cell color. In the public ARC-AGI-2 repository we counted 120 evaluation tasks holding 167 test grids; 45 tasks have more than one test input. As a sanity check we scored the laziest possible answer, copying the input grid, and it solved none of the 167. That matters because 119 of the 167 answers have the same dimensions as their input, so a model cannot score by guessing the shape. It has to find the rule.

ARC-AGI-2 has a 1,000 task training set and three calibrated eval sets of 120 tasks each (public, semi private, private), and every eval task was solved by at least two people in two attempts or fewer during testing with more than 400 members of the public (ARC-AGI-2 page). ARC-AGI-3 drops static grids for interactive games: an agent must explore, work out the goal and learn as it plays, and 100 percent means beating every game as efficiently as humans (ARC-AGI-3 page).

What Humanity's Last Exam measures

HLE was built because popular tests such as MMLU passed 90 percent accuracy and stopped separating top models. The original set has 2,500 questions across more than a hundred subjects, written by nearly 1,000 subject experts at more than 500 institutions in 50 countries (lastexam.ai). The paper appeared in Nature in 2026 (arXiv version).

HLE Diamond, released September 22, 2026, is a frozen 1,000 question subset split into 500 reasoning and 500 knowledge questions (HLE-Diamond announcement). Its authors are explicit about scope: high accuracy would show expert level performance on closed ended scientific questions, and "would not alone suggest autonomous research capabilities or artificial general intelligence." Our HLE Diamond leaderboard guide tracks those scores in detail.

So the two benchmarks sit at opposite ends. ARC removes knowledge to isolate learning. HLE piles on knowledge to find the edge of what experts know.

How each one is graded

ARC-AGI uses exact match. A test input counts only if every cell of the predicted grid is right, and the taker gets two attempts per input, the same rule for humans and machines. The ARC Prize testing policy says ARC-AGI-1 and ARC-AGI-2 runs treat the model as a direct input to output predictor with no agent harness, no code execution and no web search. ARC-AGI-3 is scored in two labeled conditions: a Standard harness that lets the model carry its own notes, and a Provider Adapter harness that uses each vendor's context features.

HLE uses an LLM as judge. The model answers and states its confidence from 0 to 100 percent, then a judge model extracts the final answer and compares it with the reference. In the official HLE repository, the judge script defaults to o3-mini and computes RMS calibration error in bins of 100 questions. Scale publishes that calibration error next to accuracy on its HLE Diamond board, and it is often the more revealing number: Gemini 4 Argon scores 53.2 percent with a calibration error of 53.3, while Claude Opus 5.5 scores 55.0 percent with 7.1. A judge can also err, which exact match cannot; see our LLM as a judge guide.

The same models on both boards

Here is how nine models score on both, using ARC Prize's verified leaderboard and Scale's HLE Diamond board (closed book), both read on November 12, 2026.

ModelARC-AGI-2ARC-AGI-3 (Standard)HLE Diamond (Scale)
Claude Fable 5.293.8%81.2%72.4%
GPT-6 Astra95.0%62.7%60.6%
Claude Opus 5.593.3%not listed55.0%
Gemini 4 Argon91.3%49.8%53.2%
Claude Sonnet 5.583.9%24.3%37.5%
Muse Spark Open57.5%0.1%28.0%
Grok 4.761.4%1.8%23.4%
Kimi K360.4%not listed22.2%
DeepSeek V4.1 Flash 101778.1%0.2%20.5%

Four things stand out.

  1. ARC-AGI-2 is close to its ceiling at the top. The four leaders on this table sit between 91.3 and 95.0 percent, and the ARC Prize human panel is listed at 100 percent. A 1.2 point gap there tells you little. HLE Diamond spreads the same four models from 53.2 to 72.4 percent.
  2. ARC-AGI-3 has the widest spread. It runs from 0.1 to 81.2 percent across these models, and several models that look solid on ARC-AGI-2 barely register on it. DeepSeek V4.1 Flash 1017 scores 78.1 percent on ARC-AGI-2 and 0.2 percent on ARC-AGI-3.
  3. The order mostly agrees, but not at first place. GPT-6 Astra leads ARC-AGI-2, while Claude Fable 5.2 leads both ARC-AGI-3 and HLE Diamond.
  4. Knowledge and learning can come apart. DeepSeek V4.1 Flash 1017 beats Muse Spark Open by 20.6 points on ARC-AGI-2 yet trails it by 7.5 points on HLE Diamond.

The HLE authors' own table at lastexam.ai uses slightly different runs: it now lists Claude Fable 5.2 at 74.3 percent and Gemini 4 Carbon at 63.6 percent, a model Scale has not scored yet. Gemini 4 Carbon posts 94.2 percent on ARC-AGI-2 and 65.1 percent on ARC-AGI-3. Quote the source with every number.

Cost, tokens and tools change the picture

ARC-AGI reports cost with every score because, in the foundation's words, brute force search could eventually solve the tasks and that "would not represent true intelligence." On ARC-AGI-2 the board lists GPT-6 Astra at $1.12 per task, GPT-6.1 Sol at $0.254 and Gemini 4 Carbon at $0.375. On ARC-AGI-3 the total run cost was $21.9K for Claude Fable 5.2 and $26.1K for GPT-6 Astra. ARC Prize uses retail API prices for these figures.

HLE reports tokens instead. At high reasoning, the launch post lists about 5,200 tokens per question for GPT-6 Astra, 8,800 for Claude Fable 5.2 and 59,200 for Gemini 4 Carbon. A model that spends ten times the tokens for a few more points may be the wrong buy.

Harness and tools move scores more than most readers expect. On ARC-AGI-3, GPT-6 Astra scores 62.7 percent in the Standard harness and 99.9 percent with the Provider Adapter. On HLE Diamond, the authors' web plus code setting lifts GPT-6 Astra from 59.9 to 82.9 percent and Claude Opus 5.5 from 54.6 to 73.9 percent. That setting is tightly specified: the HLE tool rules allow only a search tool and a fetch tool, no direct internet from the code sandbox, and a blocklist that in the version we read names 220 domains plus URL and title patterns. A "with tools" score from a vendor that skips those rules is not the same test.

Contamination and hidden test sets

ARC keeps a semi private set for testing commercial models through their APIs, under zero data retention agreements, and a private set used only for the ARC Prize competition. Because the semi private tasks travel to outside APIs, the foundation expects some leakage over time and plans a new benchmark version roughly every year (testing policy).

HLE publishes its questions but keeps a private held out set to check for overfitting. The HLE Diamond dataset on Hugging Face is gated, carries a canary string so model builders can filter it from training data, and asks users not to reshare it. Our LLM benchmarks explained guide covers contamination in general.

ARC Prize 2026 and what comes next

ARC Prize 2026 offered $2M across three tracks (ARC-AGI-3, ARC-AGI-2 and a paper prize). Submissions closed November 2, 2026 and results are due December 4, 2026 (competition page). Entries run on Kaggle without internet access and must be open source, so they answer a different question from the frontier API scores above. Scale expects HLE Diamond "to carry signal for the next 6 to 12 months."

Which benchmark should you trust?

Pick by the job you are buying a model for.

  • Expert question answering (research help, technical support, tutoring at graduate level): look at HLE Diamond accuracy and calibration error together, closed book first, then the with tools score if your product gives the model search.
  • Agents that face unfamiliar software or workflows: ARC-AGI-3 is the closest public signal for learning inside a new environment. Check which harness the score used.
  • A quick sanity check of reasoning: ARC-AGI-2 still separates mid tier and small models, but it no longer ranks the leaders.
  • Your own product: neither benchmark knows your data. Build a small test set from real tasks with one of the LLM evaluation frameworks we compare, and read our guide on how to evaluate LLM model performance.

For a broader map of boards, including where to find independent reruns, see LLM leaderboards compared. If you are weighing a science focused test as well, MMLU vs GPQA explains the older knowledge benchmarks HLE replaced.

FAQ

Which is harder, ARC-AGI or Humanity's Last Exam?

It depends on the version. Top models now score above 90 percent on ARC-AGI-2 but no higher than about 74 percent on HLE Diamond, so HLE Diamond is harder for frontier models today. ARC-AGI-3 is harder still for most models, with several scoring under 2 percent. For people the order flips: ARC tasks are built so ordinary people can solve them, while HLE questions need specialist knowledge.

Does a high score on either benchmark mean AGI?

No. HLE's authors say a high score would not alone suggest artificial general intelligence. The ARC Prize Foundation defines AGI as a system that can match the learning efficiency of humans, and its ARC-AGI-3 page says that as long as there is a gap between AI and human learning, we do not have AGI. Neither board claims a score threshold equals AGI.

Can I run ARC-AGI or HLE myself?

Yes, on the public data. ARC-AGI-2 tasks are JSON files on GitHub, and ARC Prize publishes its benchmarking code. HLE's repository has scripts that query a model and then grade it with a judge model, and the questions are on Hugging Face (HLE Diamond sits behind an access request). Scores on public data are not the same as verified scores on hidden sets, so label them clearly.

Why do ARC-AGI scores come with a cost?

Because ARC treats efficiency as part of intelligence. A system that solves tasks by trying millions of programs is less intelligent, by ARC's definition, than one that finds the rule quickly. Reporting cost per task or total run cost keeps expensive brute force from looking like progress.

Why do HLE scores differ between lastexam.ai and Scale?

They are separate runs that update at different times. On November 12, 2026 the authors' table listed Claude Fable 5.2 at 74.3 percent and GPT-6 Astra at 59.9 percent, while Scale listed 72.4 and 60.6 percent. Always name the source.