Terminal-Bench is an agent benchmark that puts a model inside a sandboxed Linux terminal and asks it to finish hard, realistic jobs: build software, configure tools, debug an environment, run science or operations workflows, then pass automated checks on the final state. Version 4.0 has 66 tasks with an 8 hour agent timeout. Version 4.1.0, released November 3, 2026, keeps those 66 tasks, hardens verifiers against reward hacking, pins container images for reproducible builds, and adds a CPU only 63 task subset. On the official 4.0 board checked November 5, 2026, Claude Code with Claude Opus 5.5 at max effort leads at 64.8% ± 3.1%.

This guide explains what Terminal-Bench measures, how 4.0 and 4.1 differ, why separate boards disagree, and how to read a score next to SWE-Bench Pro, FrontierCode and composite indexes. It is not a product review of coding agents; for dated ranking tables see our coding agent benchmark rankings.

What Terminal-Bench measures

Terminal-Bench asks a different question from patch benchmarks. SWE-Bench Pro cares whether a change makes hidden repository tests pass. FrontierCode asks whether a maintainer would merge the change. Terminal-Bench asks whether the agent can finish the job in a real shell: install dependencies, edit files, run commands, read errors and leave the environment in a state the task's verifier accepts.

The project sits under the Harbor framework and is hosted with Stanford researchers and the Laude Institute (official site, GitHub, Apache 2.0). Artificial Analysis describes the 4.0 set as spanning software, machine learning, science, operations, security, hardware and media (AA Terminal-Bench 4.0 page). Tasks can need a GPU or multiple containers; the Harbor docs recommend Modal or Daytona for large runs.

Each task ships with an English instruction, a verifier (tests on the end state) and a reference solution. A trial is pass or fail: every required check must pass. The headline metric on the official board is resolution rate (accuracy) with a 95% confidence half width. The board we read on November 5 used 330 trials per listed row, which matches 66 tasks times 5 trials.

What Terminal-Bench 4.0 changed

The 4.0 release post and v4.0.0 GitHub release (August 26, 2026) describe three kinds of change:

  1. Resource calibration. Task time, CPU and memory were recalibrated with the method in "Quantifying infrastructure noise in agentic coding evals". Every task now uses a flat 8 hour agent timeout so frontier models rarely time out for resource reasons. Remaining failures on 4.0 are described as mostly model refusals and output token limit errors.
  2. Task cleanup. Eight tasks were removed: two for saturation (every latest generation model family solved them 5/5), two for refusals, two because public solutions existed, and two for unresolved quality or platform issues. Nineteen tasks were fixed for flaky instructions, environments or verifiers.
  3. Semantic versioning. Terminal-Bench is treated as a continuous benchmark. Resource changes and a changed task set are breaking changes that need fresh trials, which is why the bump was 4.0 rather than 3.1.

Run the 4.0 dataset with Harbor:

harbor run -d terminal-bench/terminal-bench@4.0.0

What Terminal-Bench 4.1.0 changed

v4.1.0 published on November 3, 2026 (about 3:39 PM Eastern). It does not add or remove tasks. All 66 tasks were modified. The release highlights:

  • Verifier fixes and stronger isolation through RewardKit Workers, aimed at reward hacking (passing checks while missing the intent).
  • Pinned container images and lockfiles for reproducible builds.
  • A new CPU only dataset of 63 tasks for runners without GPUs.
# Full 4.1 set (still 66 tasks)
harbor run -d terminal-bench/terminal-bench@4.1.0

# CPU only subset (63 tasks)
harbor run -d terminal-bench/terminal-bench-cpu@4.1.0

There was no matching news post on tbench.ai for 4.1 when we checked November 5 (the 4.0 post remains the public writeup). The Harbor Hub package for the full set is published at dataset path terminal-bench/terminal-bench version 5. Do not mix a 4.1 run with a 4.0 leaderboard row: verifiers and images changed even though the task count stayed at 66. Launch posts now quote 4.1 figures too: Google's 57% for Gemini 4 Hydrogen is its own mini-swe-agent run with no row on the official board, as our Gemini 4 Hydrogen benchmarks check explains. Google's 65.2% for Gemini 4 Neon is an Artificial Analysis run in Antigravity CLI, also with no official row yet, as our Gemini 4 Neon benchmarks check shows.

A related research line, Hack-Verifiable Terminal Bench (HVTB), studies reward hacking on Terminal-Bench style tasks by embedding detectable hacks. It is a separate dataset from the official 4.1 hardening work; cite it only when you mean that paper.

How to read the leaderboards

Three public surfaces matter, and they are not the same experiment.

Official board (tbench.ai)

The Terminal-Bench leaderboard we loaded on November 5, 2026 still titles the package 4.0.0. Top rows (agent plus model, reasoning effort, display accuracy):

RankAgent and model (effort)Accuracy95% CI half widthTrials
1Claude Code, Opus 5.5 (max)64.8%±3.1%330
2Claude Code, Sonnet 5.5 (max)61.8%±2.9%330
3Codex, GPT-6 Astra (max)58.2%±2.8%330
3Codex, GPT-6.1 Sol (max)58.2%±3.1%330
5Claude Code, Fable 5.1 (max)57.9%±3.8%330

Source: embedded leaderboard data on tbench.ai, checked November 5, 2026. Gaps of a few points sit inside the confidence bands, so treat close rows as roughly tied.

Vals AI (Mini-SWE-agent)

Vals AI's Terminal-Bench 4.0 page, updated November 9, 2026, runs the same 4.0 task set in the Mini-SWE-agent harness and also shows cost, tokens and duration per test. As of that update, Claude Fable 5.2 (released the same day) and Claude Opus 5.5 tie for first at 65.15%, followed by Claude Sonnet 5.5 64.14%, Kimi K3.1 Pro 63.64%, Gemini 4 Carbon 63.13% and GPT-6 Astra 59.60%. The tie hides a cost gap: Vals lists Fable 5.2 at $14.56 per test against $12.04 for Opus 5.5, while Kimi K3.1 Pro costs $4.57 and Carbon $7.45 for scores within about 1.5 points. Fable 5.2 was not on the official board when we checked on November 11. Those figures are not interchangeable with Claude Code or Codex rows on the official board. For how much the harness alone moves a Terminal-Bench score, see the Harness Leaderboard, which runs nine harnesses on the older 2.1 task set.

Artificial Analysis

Artificial Analysis runs all 66 Terminal-Bench 4.0 tasks in mini-swe-agent and reports pass@1 averaged over three repeats (evaluation page). Terminal-Bench 4.0 is 10% of Intelligence Index v5 and one third of the older Coding Agent Index v2.0 composite next to FrontierCode and SWE-Atlas. For side by side agent rankings that include those composites, use our coding agent benchmark rankings.

Terminal-Bench vs SWE-Bench Pro, FrontierCode and GDPx

BenchmarkCore questionPublic task count (current)Typical pass signal
Terminal-Bench 4.xDid the shell job finish and pass verifiers?66 (4.0 and 4.1 full); 63 CPU only in 4.1Every task test passes
SWE-Bench Pro V2Did the patch pass hidden repo tests?642 public; HARD-51 subsetHidden tests pass, no regressions
FrontierCodeWould a maintainer merge this?Private Main / Extended (and 2.0)Blockers pass; rubric score if they do
GDPxCan the agent finish long professional work across domains?206 mostly privatePass@1 over 3 attempts, equal domain weights

Use Terminal-Bench when you care about end to end terminal competence (install, configure, debug, operate). Use SWE-Bench Pro when the product is repository patches. Use FrontierCode when mergeability and review quality matter. Use GDPx when you want long horizon professional domains beyond the shell. For how to structure your own eval instead of quoting a public board, see how to evaluate LLM model performance.

Terminal-Bench-Science

Terminal-Bench-Science is a sibling suite from the same community: 70 research workflow tasks across life, physical, mathematical, engineering and earth sciences. Artificial Analysis weights it at 15% of Intelligence Index v5. It is not the same dataset as Terminal-Bench 4.x; a Science score does not substitute for a 4.0 or 4.1 shell score.

Limits to keep in mind

  1. Harness dominates. Claude Code, Codex and Mini-SWE-agent produce different numbers for the same model. Always name agent, model and effort.
  2. Version and date. 4.0 removed eight tasks and recalibrated resources. 4.1 reworked verifiers and images without changing the count. A 4.1 private run is not a 4.0 leaderboard score.
  3. Confidence bands. Official top rows sit about three points apart with ±3% half widths. Small gaps are weak evidence.
  4. Cost and tokens. Vals and the official board both show large spend and token counts on long tasks. A higher score can be expensive; compare cost columns when they exist.
  5. Reward hacking. Passing verifiers is not the same as doing the intended work. 4.1 and research such as HVTB exist because agents exploit checks.
  6. Sibling confusion. Terminal-Bench-Science and the CPU only subset are related but separate denominators.
  7. Your workload still wins. Public tasks are a shortlist. Validate on the shells, repos and runbooks you actually operate.

FAQ

What is Terminal-Bench?

Terminal-Bench is an open agent benchmark that scores whether a model, wired to a terminal harness, can complete hard real world tasks in a Linux sandbox so that automated verifiers pass. The current full set has 66 tasks (versions 4.0 and 4.1).

What is the difference between Terminal-Bench 4.0 and 4.1?

4.0 (August 2026) removed eight weak or saturated tasks, fixed nineteen others and set an 8 hour timeout. 4.1.0 (November 3, 2026) keeps 66 tasks, hardens verifiers, pins images and adds a 63 task CPU only package. Rerun trials after moving between them.

Who leads the Terminal-Bench 4.0 leaderboard?

On the official board checked November 5, 2026, Claude Code with Claude Opus 5.5 (max) leads at 64.8% ± 3.1%. Vals AI's Mini-SWE-agent board, updated November 9, has Claude Fable 5.2 and Opus 5.5 tied first at 65.15%. Independent Artificial Analysis runs use their own harness and repeats.

Is Terminal-Bench the same as SWE-bench?

No. SWE-bench and SWE-Bench Pro grade repository patches against tests. Terminal-Bench grades shell workflows against task verifiers. An agent can be strong on one and weak on the other.

How do I run Terminal-Bench myself?

Install the Harbor CLI, then run harbor run -d terminal-bench/terminal-bench@4.1.0 (or @4.0.0 to match older boards). Use terminal-bench-cpu@4.1.0 when you need the CPU only subset. Follow Harbor's docs for Modal or Daytona if tasks need GPUs or many containers.

What is Terminal-Bench-Science?

A separate 70 task research workflow benchmark from the Terminal-Bench and Harbor community. It appears in Artificial Analysis Intelligence Index v5 as its own component and should not be quoted as a Terminal-Bench 4.x score.