The Harness Leaderboard is a Hugging Face Space by Joel Niklaus that holds the model fixed and swaps the coding agent harness around it, so you can see how much the harness alone moves a score. On version 0.1.3 (November 10, 2026), averaged over three open models, OpenCode leads at 52.0%, with mini-swe-agent, Kimi Code and Hermes Agent statistically tied, and OpenClaw last at 27.0%. That 25 point spread is the headline. The fine print matters just as much: the best harness changes with the model, and several results on the board are still waiting for reruns.

This guide explains what the board measures, what the current numbers say, how to read its noise and cost sections, and which caveats to quote alongside any ranking. It is a benchmark explainer, not a product review. If the word harness is new to you, drydock.si has a plain explainer on what an agent harness is.

What the Harness Leaderboard measures

Most coding leaderboards rank a model, or a model inside one harness. The Harness Leaderboard asks the opposite question: given one open model, which harness should you run it in? According to the Space README, every row uses the same model weights, the same reasoning effort, the same task text, time limit, internet access and scoring. Only the harness changes. Each harness keeps its own system prompt, tools, history compaction, retries and subagents as shipped.

The setup on v0.1.3:

  • Nine harnesses: Claude Code, Codex, Pi, OpenCode, Hermes Agent, mini-swe-agent, Kimi Code, Qwen Code and OpenClaw, each pinned to a version.
  • Three models: GLM-5.3-Flash and DeepSeek-V4.1-Flash at low and high reasoning effort, plus Flint at its default effort.
  • Two benchmarks: Terminal-Bench 2.1 (87 of its 89 tasks; two QEMU tasks are excluded because they need a patched image) and DeepSWE v1.1 (111 of 113 tasks; two are excluded for broken reference solutions or graders).
  • Three attempts per task, run through Harbor on Hugging Face Inference Providers, HF Sandbox and HF Jobs.

The headline score is the macro average resolve rate across the two benchmarks with a 95% confidence interval. A paired bootstrap marks harnesses whose gap to the leader could be zero as tied. Every trajectory is public in a Hugging Face bucket. A metering proxy inside each sandbox enforces the reasoning effort on every model call and counts tokens, so cost and time are measured outside the harness.

Two rules stand out. Cheating counts as a fail: looking up a benchmark's published solution, fetching the upstream code a task was cut from, or calling another model all fail the trial, and every trajectory is checked with Hugging Face's atif-scan before flagged passes are read by hand. And FeatureBench, a third benchmark, has been unscored since October 25 after its task images turned out to hold the reference solutions.

Current Harness Leaderboard rankings (v0.1.3)

We pulled these from the board's public data on November 10, 2026 at version 0.1.3. All models, all efforts, macro average:

HarnessScore95% intervalTied with leader?
OpenCode 2.2.652.0%49.1 to 54.8Leader
mini-swe-agent 2.4.650.7%47.8 to 53.6Yes
Kimi Code 2.2.050.5%47.6 to 53.4Yes
Hermes Agent v0.22.150.1%47.2 to 52.9Yes
Claude Code 2.1.30049.2%46.4 to 51.9No
Pi 1.2.048.7%45.9 to 51.5No
Qwen Code 0.25.244.7%42.0 to 47.4No
Codex 0.162.144.5%41.7 to 47.2No
OpenClaw 2026.10.427.0%24.6 to 29.5No

Read the top of that table as a group, not a podium. Four harnesses sit within 2 points, and Claude Code and Pi are only a little further back. The real separation is at the bottom: Qwen Code and Codex trail by about 7 points, and OpenClaw by 25.

An earlier draft write up in the Space's repo has mini-swe-agent first at 51.4%, from before OpenCode 2 replaced OpenCode 1.19.0 on November 9 and moved OpenCode to the top. That is the kind of change a young board makes, so always quote the version number with a score.

The best harness depends on the model

Split the board by model and the winner changes.

GLM-5.3-Flash (both efforts). OpenCode 62.4%, mini-swe-agent 62.2%, Pi 61.6%, Kimi Code 61.5%, Claude Code 60.8% and Hermes Agent 60.4% are all marked tied. Qwen Code (54.6%), Codex (47.4%) and OpenClaw (40.0%) are not.

DeepSeek-V4.1-Flash (both efforts). Hermes Agent leads at 71.0%, with OpenCode, Claude Code, Codex, mini-swe-agent and Kimi Code tied. Codex does far better here (68.9%) than on GLM-5.3-Flash, which shows how much a harness and model pairing matters.

Flint (default effort). Scores are much lower for everyone. OpenCode 22.9%, Kimi Code 21.7% and mini-swe-agent 21.4% are tied at the top, and OpenClaw manages 5.6%.

mini-swe-agent, the smallest harness on the board, is tied for first on all three models. That matches an independent study, What Does a Harness Buy? Tokens, Mostly (October 2026), which found Claude Code and mini-SWE-agent equivalent within five points on 447 SWE-bench Verified tasks.

Harness vs noise: how much is real

The board's Noise section asks a simple question. If you run the same harness twice, how often does a task flip between pass and fail? Then, how often does it flip when you swap the harness instead?

On v0.1.3, across all models and efforts:

  • Rerunning the same harness flips 23.5% of tasks.
  • Swapping the harness flips 28.5%.
  • Changing the reasoning effort flips 27.0%.
  • Changing the model flips 33.1%.

So the harness does matter, but most of the task level churn you see between two harnesses would also show up between two runs of one harness. The arXiv study found the same thing on a hard 45 task subset of SWE-bench Verified: swapping the harness flipped 13% of tasks, and so did a plain rerun. It also warned that 45 tasks catch a 13 point gap only half the time, while 447 tasks resolve about 5 points. That is why three attempts and confidence intervals are the most useful part of this board.

Reliability over three attempts

For unattended work, solving a task every time matters more than solving it once. The Reliability section shows, per harness, the share of tasks solved on all three attempts and the share solved on at least one.

Across all models, OpenCode solved 35.6% of tasks on all three attempts and 68.0% on at least one. mini-swe-agent was at 32.5% and 67.9%, Claude Code at 32.1% and 64.9%, Codex at 26.8% and 61.5%, and OpenClaw at 15.0% and 41.2%. The gap between those two numbers is roughly a third of all tasks for most harnesses, which tells you how often an agent gets lucky.

What each harness costs per task

Every trial is priced at the cheapest checked provider's list price (Flint uses GPU hour pricing), so cost on the board does not depend on which provider served a trial. Across all models and efforts:

HarnessCost per trialCost per solved task
Pi$0.097$0.29
Qwen Code$0.117$0.34
Claude Code$0.145$0.52
OpenCode$0.149$0.45
Codex$0.169$0.71
Hermes Agent$0.175$0.65
mini-swe-agent$0.180$0.54
Kimi Code$0.188$0.63
OpenClaw$0.196$1.43

Pi costs about two thirds of what Claude Code costs per trial and scores 0.5 points lower. OpenClaw is the most expensive per solved task by a wide margin. The arXiv paper explains where these gaps come from: the system prompt and tool schemas each harness sends with every step set the bill, scaled by the number of steps. In its tests cost per task differed by up to three times across harnesses on the same model.

One warning from the board's own changelog: since November 7 every run passes each model's real context window and longest reply to the harness, and OpenCode 2 is the first result on the board that ran with those limits. Older results ran with each harness's defaults (Pi, for example, assumed a 128,000 token window and compacted early, which kept its calls cheap). So cost per call is not yet comparable across every row.

Caveats before you quote a harness ranking

The changelog is unusually candid. These are the items most likely to change the table:

  1. OpenClaw's last place is partly a settings problem. Its board results ran version 2026.10.4, which asked for at most 8,192 tokens per reply because no output limit was set. On Flint and DeepSeek-V4.1-Flash that cap cut replies off and often ended runs. A rerun at 2026.11.1 with each model's limits is in progress.
  2. Images were not shown to models that read them. Six harnesses never passed image files to GLM-5.3-Flash or DeepSeek-V4.1-Flash. Reruns of the affected trials are queued, and Hermes Agent reruns in full.
  3. Kimi Code on Flint was told the wrong context window (1,000,000 tokens instead of 262,144). The 196 Flint trials that outgrew the real window run again.
  4. DeepSWE had internet access until October 23. Hundreds of passes looked up answers online. Version 0.1.3 replaced 326 of them with offline reruns; some DeepSeek results for Hermes Agent, Kimi Code, OpenClaw and Qwen Code still count as they ran until their full reruns land.
  5. Harnesses ran on different days, at whatever speed providers delivered that day, while the time limit per task stayed fixed.
  6. Only open models are tested. Claude Code and Codex are normally paired with Anthropic and OpenAI models. Their scores here say how they drive GLM, DeepSeek and Flint, not how they drive their home models.

How it compares with other coding leaderboards

The Artificial Analysis Coding Agent Index and the official Terminal-Bench board score model and harness pairs, mostly frontier models, so a single row mixes both effects. Vals AI fixes one harness (Mini-SWE-agent) for every model. Our coding agent benchmark rankings put those boards side by side, and our Terminal-Bench explainer covers versions 4.0 and 4.1, which are newer than the 2.1 set this board uses. FrontierCode and SWE-Bench Pro add patch quality and mergeability views.

Use the Harness Leaderboard for one decision: you have picked an open model and want to know which harness to wrap it in. Use the others to pick the model. For how all these boards differ in method, see LLM leaderboards compared.

How to use the board

  1. Pick your model and effort in the filters instead of reading the averaged view.
  2. Look at the tie marks first, then the score.
  3. Check Reliability if the agent will run without a person watching.
  4. Check cost per solved task, not cost per trial.
  5. Read the changelog's Unreleased section for reruns that affect your harness.
  6. Open a few public trajectories for your harness to see how it fails.

FAQ

What is the Harness Leaderboard?

A Hugging Face Space by Joel Niklaus, first released November 5, 2026, that runs nine coding agent harnesses on the same open models, reasoning efforts and tasks, with three attempts per task, confidence intervals and public trajectories. It measures which harness to run a given model in.

Which coding agent harness is best on the Harness Leaderboard?

On v0.1.3, averaged over all models, OpenCode leads at 52.0%, with mini-swe-agent, Kimi Code and Hermes Agent statistically tied. The leader changes by model: Hermes Agent tops DeepSeek-V4.1-Flash, and six harnesses tie on GLM-5.3-Flash.

Which benchmarks does the Harness Leaderboard use?

Terminal-Bench 2.1 (87 tasks) and DeepSWE v1.1 (111 tasks), run through the Harbor framework. FeatureBench was dropped from scoring on October 25, 2026 after its task images were found to contain reference solutions.

Does the harness matter more than the model?

No. On the board, changing the model flips 33.1% of tasks and changing the harness flips 28.5%, against 23.5% for a simple rerun. The harness matters, but less than the model, and much of its apparent effect is noise.

Why does OpenClaw score so low?

Partly because its board results ran with an 8,192 token reply cap that cut off replies on two of the three models. A rerun with each model's real limits is in progress, so expect its numbers to change.

Is the Harness Leaderboard free to use?

Yes. The Space is public under the Apache 2.0 license, and all trajectories and result records are in a public bucket. Signing in with Hugging Face is optional for viewing.