SWE-Bench Pro is Scale AI's coding agent benchmark for long horizon software engineering: given a real repository and an issue, an agent must produce a patch that passes hidden tests. It sits beside the older Princeton SWE-bench family (including SWE-bench Verified) and is what OpenAI recommended after it stopped reporting Verified scores in February 2026. On September 22, 2026, Scale released SWE-Bench Pro V2, a cleaned public split of 642 tasks plus a HARD-51 subset and a locked evaluation protocol. If you're a person facing a timed coding assessment for a job, see how to take a coding test instead. For how the SWE-bench family differs from a knowledge exam, see MMLU vs SWE-bench.
This guide explains what Pro measures, how the public, private and held out sets differ, what V2 changed, and how to read a launch day score without mixing versions or mistaking memorization for skill.
What SWE-Bench Pro is
Scale describes SWE-Bench Pro as a harder follow on to SWE-bench, built for realistic, multifile, enterprise style work rather than short algorithmic puzzles. The ICML 2026 paper reports 1,865 problems from 41 repositories, with public problems from 11 repositories and private evaluation sets from additional repositories (PMLR paper; Scale paper page).
Scale's public overview breaks the full collection into three parts (public leaderboard page):
| Split | Size (as published) | Access | Role |
|---|---|---|---|
| Public set | 731 instances (V1) from 11 open source repos | Public dataset and leaderboard | Main open comparison surface |
| Commercial / private set | About 276 instances from proprietary startup codebases | Results only; code not public | Cleaner generalization check |
| Held out set | 858 instances | Not published on public boards | Reserved for later analysis |
Public instances use strong copyleft licenses such as GPL. Scale's stated reason is to discourage inclusion in model training data. That is a legal and policy deterrent, not a guarantee of zero contamination.
The task format matches the SWE-bench idea: the agent sees the issue and the prefix codebase, not the hidden tests. A resolve counts only when the tests pass after the patch is applied. Pro tasks are designed to be longer horizon than classic SWE-bench items, often multifile patches that can take a human engineer hours or longer, according to Scale's paper abstract.
SWE-bench Verified vs SWE-Bench Pro
People still type "swe-bench verified vs pro" because both appear on launch slides. They are related but not interchangeable.
SWE-bench Verified is a 500 task human filtered subset of the original Princeton SWE-bench. It became the default coding score for a while. On February 23, 2026, OpenAI published Why we no longer evaluate SWE-bench Verified and stopped reporting it for frontier launches. Two findings drove that decision:
- Flawed tests on hard tasks. In an audit of 138 Verified problems that models often failed, at least 59.4 percent had material issues in tests or descriptions (narrow tests that demand a specific implementation detail, wide tests that check unspecified behavior, or other defects).
- Training exposure. Frontier models could reproduce gold patches or verbatim problem details for some tasks, which is a strong sign that solutions or near copies appeared in training data.
In the same post, OpenAI said it had shifted to reporting the public split of SWE-Bench Pro and recommended that other developers do the same. It noted that Pro still had some contamination hits under their probe, but those cases were rarer and less egregious than on Verified, and no model produced a complete verbatim gold patch in their setup. That is an improvement, not a claim that Pro is uncontaminated forever.
For the broader map of coding and knowledge benchmarks, see LLM benchmarks explained. For how leaderboards mix vendor and independent runs, see LLM leaderboards compared.
What SWE-Bench Pro V2 changed
Scale released SWE-Bench Pro V2 on September 22, 2026, co-developed with Reflection (Scale Labs blog; GitHub release note). The Hugging Face default config is now V2; the original public set remains available as config v1 / tag v1.0 (Hugging Face dataset).
Documented V2 changes:
- 642 public tasks across the same 11 repositories, down from 731 after 89 invalid tasks were removed.
- HARD-51: 51 tasks from those 642 that Scale found most challenging for frontier systems.
- Task instructions, verifiers and container images were refreshed after human expert and agentic review for reward hacking, underspecification, solution leakage and related quality issues.
- 211 tasks got better dependency support for open source harnesses.
- Evaluation hardening: sanitized images (no fixing commit or stray git state), a network locked agent phase (model endpoint only), and regrading on a pristine image so in sandbox cheats fail on a clean grader.
- Scale reports that 23 contracted engineers spent 1,897 hours on corrections, with blind reimplementation checks for rewritten instructions.
V2's public leaderboard also shows near ceiling Pass@1 numbers for several frontier systems on the full 642 task suite (Scale's post lists Claude Opus 5 at 99.4 percent, Kimi K3 at 97.7 percent, and GPT-Astra at 96.9 percent on the public set). Scale itself flags the gap between public and private results and treats training time exposure of public repos and fixing commits as a plausible explanation when public scores sit near the ceiling. The private set is the cleaner readout for generalization. HARD-51 exists because the full public set no longer separates the top agents well once grades are trustworthy.
Do not mix V1 and V2 percentages. A 731 task V1 score and a 642 task V2 score are different denominators and, after the refresh, different task quality.
How scoring and harnesses work
A Pro result is almost always resolve rate / Pass@1: the share of instances whose hidden tests pass after one agent attempt under a stated harness and budget. That number depends on more than the base model:
- Harness (tools, system prompt, timeouts). Scale and Reflection's V2 study ran open weight models through mini-swe-agent, Pi and OpenCode under a locked budget and found that accuracy stayed similar for the strongest models while token and time cost swung with the harness (blog section on harnesses).
- Network and environment. V2's locked protocol blocks code hosts and package indexes during the agent phase. A looser setup can look stronger for the wrong reasons.
- Who ran it. Vendor slides, Scale's public board and third party aggregators such as Artificial Analysis are not always the same setup. Prefer the board and config named in the citation.
If you are building your own coding agent eval rather than quoting a board, use the same discipline you would for any LLM app: fixed tasks, recorded harness, and more than one metric. Our how to evaluate LLM applications playbook covers that workflow. For a benchmark that also grades whether a maintainer would merge the patch, see FrontierCode explained.
How to read a SWE-Bench Pro claim
- Name the version. V1 public (731), V2 public (642), HARD-51, private/commercial, or held out.
- Name the split. Public scores can be inflated by training exposure; private scores are the harder generalization check when available.
- Name the harness and limits. Turns, tokens, tools and network policy change outcomes and cost.
- Compare like with like. Do not rank a V2 Pass@1 against a Verified score or against Humanity's Last Exam. Different skills, different tests. See Humanity's Last Exam for the hard academic exam, and MMLU for saturated multiple choice knowledge.
- Treat near ceiling public scores as weak differentiators. Prefer HARD-51, private results or independent trackers when public numbers bunch together.
- Still test on your repos. Benchmarks shortlist agents; your failing tickets decide.
For a dated table that puts SWE-Bench Pro V2 next to FrontierCode, Terminal-Bench and the Artificial Analysis index, see our coding agent benchmark rankings.
Fresh, contamination resistant boards such as LiveBench are useful cross checks for general capability, but they do not replace a coding agent suite when your product is about patches and tests.
Who should use which number
| Audience | Prefer | Why |
|---|---|---|
| Model labs publishing a launch | V2 public + private (when available), with harness named | Matches current Scale protocol; private resists memorization |
| Buyers comparing coding agents | Independent or Scale private/HARD-51, not a lone public percentage | Public V2 can sit near the ceiling for several frontiers |
| Researchers reproducing results | Hugging Face configs (default / hard / v1) and the open Harbor tasks | Exact task IDs and images are published for V2 |
| Product teams | A small internal suite of real issues | Closest to your stack, licenses and review process |
FAQ
What is SWE-Bench Pro?
SWE-Bench Pro is Scale AI's benchmark where coding agents must fix long horizon issues in real repositories so that hidden tests pass. It includes public, private and held out splits and was published at ICML 2026.
How is SWE-Bench Pro different from SWE-bench Verified?
Verified is a 500 task filtered subset of the original Princeton SWE-bench. Pro is Scale's harder, longer horizon suite with public and private sets. OpenAI stopped reporting Verified in February 2026 and recommended Pro's public split instead.
What is SWE-Bench Pro V2?
V2 is the September 22, 2026 refresh of the public set: 642 validated tasks (89 invalid V1 tasks removed), a HARD-51 subset, refreshed images and verifiers, and a network locked evaluation protocol with pristine image regrading.
Is SWE-Bench Pro contaminated?
Public repositories and fixing commits can still appear in training data. Scale and OpenAI both treat public scores as imperfect; Scale emphasizes the private set when public results approach the ceiling. Pro was less contaminated than Verified in OpenAI's February 2026 probes, not contamination proof.
What does a SWE-Bench Pro score mean?
Usually Pass@1 resolve rate under a named harness: the fraction of tasks whose hidden tests pass after one attempt. Always check whether the number is V1 or V2, public or private, and which tools and network rules were allowed.
Should I still look at SWE-bench Verified?
Only as historical context. For frontier comparisons in 2026, prefer SWE-Bench Pro (ideally V2 and private/HARD-51) plus your own repository tests. Verified remains useful for understanding older papers and charts.
