The SO-Arena benchmark is a new test for System One models, the decision models that return a probability for each allowed answer in a single pass instead of writing text. It scores 21 models on 2,919 tasks across 11 domains, and its headline finding is that the best System One model on capability, Fastino's GLiDE at 70.2, ranks only seventh of 18 on reliability. OpenAI's GPT-6 Sol (82.3) and GPT-6 Luna (79.7) still lead overall, but they take seconds per task where open decision models answer in well under a third of a second.
This guide explains what SO-Arena measures, what the current numbers say, where the scores are thin, and how to run it on your own model. If decision models are new to you, our guide to picking an LLM judge model covers how teams use them as cheap judges.
What SO-Arena is
SO-Arena was posted to arXiv on November 11, 2026 by Lichen Li, Ziping Dong, Sicheng Feng, Qi Li and Xinchao Wang (paper). On November 14 it was among the most upvoted entries on Hugging Face's daily papers feed, with 84 votes (Hugging Face papers page). The code is on GitHub under the MIT license, and the data sits in a Hugging Face dataset.
The paper borrows its framing from TypeSafe, which launched Jev in September as a "System One" model that gives up text generation and returns typed answers (TypeSafe launch post). Every SO-Arena task is posed in that interface: pick one option, answer a yes or no question, or give a grade on a fixed scale. Free text answers are excluded.
The benchmark draws on 40 public datasets and environments, including GoEmotions, MATH500, GPQA Diamond, Mind2Web and the Berkeley Function Calling Leaderboard. The authors kept the source labels, reviewed them by hand and with LLMs, and dropped items whose reference answer looked wrong.
The 2,919 tasks in 11 domains
The tasks fall into five capability families and 11 domains. There are 2,300 static decisions, 282 tool use episodes run inside the BFCL and ToolSandbox simulators, and 337 text based web tasks.
| Family | Domains (tasks) |
|---|---|
| Semantic understanding | Sentiment and Emotion (200), Intent and Routing (300) |
| Knowledge and reasoning | Math (200), Code Understanding (200), Scientific Reasoning (200), Knowledge and Commonsense (200) |
| Safety and policy | Safety (500) |
| Evidence assessment | Retrieval and Reranking (300), Task Completion Verification (200) |
| Agent task completion | Tool and API Use (282), Text based Web Interaction (337) |
Nine domains have easy, medium and hard tiers, and Math adds 30 very hard problems. Sentiment and Knowledge are untiered. The tiers are set per domain, so a hard Safety item and a hard Math item are not equally hard.
The agent tasks are the unusual part. In tool episodes the model picks each action from a menu of public options and the simulator executes it, so one wrong pick can end the episode. In Task Completion Verification the model plays the judge instead, deciding whether a recorded run met its goal, including runs that reach the same end state by different routes.
How the leaderboard score works
The leaderboard score does not use all 2,919 tasks. It uses a panel of 1,232: the hardest tier of every graded domain (hard plus very hard for Math) and every task in the two untiered domains. Each domain's score is the mean credit on its panel tasks times 100, and the 11 domains count equally (scoring script).
That choice makes the board sensitive to hard cases, which is the point. It also means some domains rest on very few items. When we rebuilt the panel, Task Completion Verification had only 20 panel tasks, so a single item moves that domain by 5 points and the overall score by almost half a point. Code Understanding has 70 panel tasks and Scientific Reasoning 80, while Sentiment and Knowledge have 200 each.
SO-Arena leaderboard: current results
These are the paper's scores, which we recomputed from the released per task credits.
| Model | Type | Score |
|---|---|---|
| GPT-6 Sol (medium effort) | System Two reference | 82.3 |
| GPT-6 Luna (medium effort) | System Two reference | 79.7 |
| GLiDE (Fastino, 27B) | Open weight System One | 70.2 |
| KnowLine-31B (PelaAI) | Open weight System One | 70.2 |
| Sieve-31B-v3 (Sthanika AI) | Open weight System One | 67.7 |
| Decider-27B (Perplexity) | Open weight System One | 66.7 |
| Kev-27B | Open weight System One | 61.4 |
| Jev API 1.14 (TypeSafe) | Closed decision API | 60.4 |
| StartLux Instant (4B) | Open weight System One | 59.0 |
| OpenAI Decisions | Closed decision API | 53.6 |
| Laya (0.4B) | Open weight System One | 30.9 |
GLiDE and KnowLine tie at 70.2 on the printed table. Unrounded, GLiDE leads by 0.02 points, which is less than one task. Read them as joint leaders among System One models.
The two closed decision APIs land mid table. Jev scores 60.4 and OpenAI Decisions 53.6. Part of the OpenAI gap is refusals: it declined four static tasks and twelve environment episodes, and each refusal scored zero. The paper's example is a Safety item whose input contains jailbreak text the model was only asked to classify.
On November 13 the authors added Google's Gemini Decision 1 to the project page, run after the paper on the same release. It scores 61.2, just behind Kev-27B and just ahead of Jev, with a reliability of 75.0. That is the first independent benchmark run of Gemini Decision 1 we have seen.
Why reliability is scored separately
SO-Arena's second score asks whether a decision holds up when the options change. It has two parts:
- Order stability (S). Each item is shown in its original order and in 12 reorderings. S measures how often the model keeps the same answer, rescaled so random picks score 0 and perfect agreement scores 1.
- Null discrimination (N). Each item is shown twice, once as is and once with the correct option removed. N measures how well the model's probability that no option is correct separates the two copies, from 0 at chance to 1.
Reliability R is the square root of S times N, shown on a 0 to 100 scale. A model that is stable but never notices a missing answer still scores low.
The ranking changes a lot. JevAny-27B leads on reliability at 84.1 and the Jev API follows at 83.5, though they rank only eighth and sixth of the 18 System One models on capability. GLiDE, first on capability, is seventh on reliability at 77.6, mainly because its null discrimination (0.64) trails the leaders (about 0.78). OpenAI Decisions is one of four models the paper calls sensitive to option order, at 0.68.
The paper's broader finding is useful for anyone choosing a model. Order stability tracks capability closely: models mostly flip answers on questions they get wrong anyway. Null discrimination does not. Kev-27B, the Jev API and JevAny-27B all beat the four best performing System One models at spotting a missing answer. If your pipeline feeds a decision model options that may not include the right one, test that case directly. Our guide to LLM confidence scores covers why a probability is not the same as being right.
Speed: where System One models win
Every open weight System One model answered in under 0.3 seconds per task on one NVIDIA H100, measured over 1,109 timed tasks with three timed passes each. Laya averaged 6 milliseconds, GLiDE 113 and Sieve-31B-v3 265. The closed APIs were timed end to end over the network: the Jev API at about 249 milliseconds, OpenAI Decisions at about 302, and the GPT-6 Sol reference at about 2.8 seconds per task. Gemini Decision 1 averaged about 608 milliseconds with four requests in flight, and the authors note its speed varied during the day.
So the tradeoff is real. GLiDE gives up about 12 points to GPT-6 Sol overall and answers roughly 25 times faster on one local GPU, although the GPT-6 time also includes the network trip.
Where System One matches System Two
The paper compares GPT-6 Luna with the best 4B model (StartLux Instant) and the best 27B model (GLiDE) and finds three groups:
- Direct judgments. In Knowledge, Sentiment and Retrieval, System One models score close to Luna, and neither a bigger backbone nor thinking helps much.
- Grounded decisions. In Intent and Routing, Safety, Task Completion Verification, Tool Use and Web, GLiDE comes within 4 points of Luna, while StartLux Instant trails by up to 29. A newer, larger backbone is what closes the gap here.
- Multistep derivation. In Math, Scientific Reasoning and Code, both System One models trail far behind. Thinking helps most in Math and Scientific Reasoning: letting StartLux Auto think on every request lifts its Math score from 50.8 to 92.5.
In plain terms, a decision model is a sound pick for routing, classification, safety labels and checking whether an agent finished its job, but not for anything that needs worked steps.
How SO-Arena compares with other decision model boards
SO-Arena is the third public yardstick for this model class. The community Jev Decision Index ranks 275 decision models on private tests, and Artificial Analysis publishes its own Decision Index v2, which folds calibration into the headline number.
SO-Arena differs in three ways. Its test items are public, which helps anyone reproduce a score but means future models could train on them (the dataset carries a canary string to discourage that). It includes interactive tool and web episodes, which the other two do not. And it keeps reliability as a separate score rather than blending it into one number. A GLiDE model sits near the top of all three, which is a reasonable sign the boards agree on the leaders. For how these fit with general model boards, see our LLM leaderboards compared guide.
How to run SO-Arena on your own model
The repo needs Linux on x86 and Python 3.12. The steps from the README are:
- Clone the repo, create a virtual environment and install
huggingface_hub. - Run
./jev fetch-datato pull the pinned data, then./jev verify,./jev setupand./jev doctor. - Point
JEV_ENDPOINTat your model server and run./jev run --adapter adapters:http. - Run
./jev scoreto get every task's credit, domain and tier scores, and the leaderboard score once the run covers the 1,232 panel tasks.
Use --domain, --tier or --limit for a quick smoke test first. The reliability and latency runs live in their own folders.
Check the licenses before commercial use. The code is MIT and the authors' annotations are CC BY 4.0, but every item keeps its source's terms, and the dataset's license file notes that several sources are non commercial. The GPQA Diamond and Mind2Web items ship in a password protected archive because their sources ask that they not be published in plain text.
FAQ
What is the SO-Arena benchmark?
SO-Arena is a benchmark released on November 11, 2026 that tests System One decision models on 2,919 tasks in 11 domains, from sentiment and routing to math, tool use and web navigation. It reports a capability score and a separate reliability score.
Which model leads SO-Arena?
GPT-6 Sol leads overall at 82.3, followed by GPT-6 Luna at 79.7. Among System One models, GLiDE (70.2) and KnowLine-31B (70.2) are tied at the top, with GLiDE ahead by 0.02 points before rounding.
What does reliability mean in SO-Arena?
Reliability combines two tests: whether a model keeps its answer when the options are shuffled, and whether it notices when the correct option has been removed. JevAny-27B scores highest at 84.1, while GLiDE ranks seventh at 77.6.
Is SO-Arena a good benchmark for choosing a judge model?
It is a useful starting point, especially the Task Completion Verification domain, but that domain has only 20 tasks on the leaderboard panel. Treat it as a shortlist and then test candidates on a few hundred labeled examples from your own task.
Can I use SO-Arena data commercially?
The code is MIT licensed and the authors' annotations are CC BY 4.0, but each item keeps its original source's license, and several sources are non commercial. Check the license file in the dataset before any commercial use.



