The Jev Decision Index is a community leaderboard on Hugging Face that runs open copies of TypeSafe's Jev decision model, plus Jev itself, through the same 110,201 requests and ranks them on private tests nobody can train on. On the November 6, 2026 data, REAX 1.3 (73.98) and GLiDE 27B (73.37) share first place, and Jev 1.14 sits 55th of 275 at 60.24. It is unofficial, and it is not the same thing as the Artificial Analysis Decision Index, which shipped its own v2 on November 8.
This guide explains what the board measures, how the Full score is built, what the current numbers say, and how to tell the two "Decision Index" leaderboards apart. It is a benchmark explainer, not a model review. If decision models are new to you, start with our guide to picking an LLM judge model, which covers Jev and the OpenAI Decisions API as graders.
What the Jev Decision Index is
TypeSafe launched Jev in early access on September 15, 2026, as a "System One" model that gives up text generation and returns typed answers with a probability on every option (TypeSafe launch post). The weights and the training method, which TypeSafe calls Reinforcement Learning for Calibrated Decisions, were never released. Within days, dozens of teams published models that copy the interface: a state goes in, a set of typed questions goes in, and one probability per option comes out.
The Jev Decision Index Space tracks those copies. Its README describes it as unofficial, community maintained and not affiliated with TypeSafe. The Space has two tabs. News is a grid of reproductions, explainers and prior art claims. Index is the leaderboard, and that is what people mean by the phrase. A reproduction kit on GitHub rebuilds the public test suite so anyone can score a model locally or as one Hugging Face Job.
The current edition is Decision Index 0.4.1, dated October 21, 2026. The data files we downloaded were generated on November 6 and list 275 models on the text board, including Jev, plus 78 entrants on a separate Vision board.
How the Full score works
Since edition 0.4 (October 19), the ranking rests only on private data. According to the board's changelog, the Full score is:
- 60% private tests of the same skills as the public benchmarks, built from fresh data
- 40% private decision tasks from new domains
Each private part is mapped onto the public scale using untuned stock models as the yardstick. A model that sits a given distance above the stock models' average on a private part gets the score that sits the same distance above their average on the public index. The methodology file lists 21 stock bases for that mapping.
The public benchmarks still run and still show next to every model, but they count for nothing. In edition 0.3 they were 20% of the score. The maintainers dropped them because entrants can train on public data, and the kit's README says a model whose public score sits far above its private ones "gets a closer look". If the checks find that a model's training data overlapped a private test, that test is dropped for that model and its entry says so, without naming the test.
The public suite
The public index is 37 benchmarks in five areas. Every score is chance corrected, so 0 means random guessing and 100 means perfect, and an unanswered request counts as wrong.
| Area | Weight | Examples |
|---|---|---|
| Knowledge & Reasoning | 25.8% | GPQA Diamond, MMLU-Pro, HLE, BBH, GSM8K |
| Language Understanding | 25.8% | ANLI, HellaSwag, ContractNLI, RAGTruth |
| Retrieval & Classification | 20.0% | BANKING77, CLINC150, BRIGHT, Amazon ESCI |
| Tools & Automation | 18.3% | BFCL, ToolRet, API Bank, When2Call |
| Arts & Human Taste | 10% | New Yorker captions, Humicroedit, POP909 |
Arts is fixed at 10%. The other four areas split the rest by the square root of their benchmark count, and gold benchmarks such as MMLU-Pro and BFCL weigh 1.2 inside their area. If some names are unfamiliar, our explainers on MMLU vs MMLU-Pro, Humanity's Last Exam and tool calling benchmarks cover several of them.
Rules every entrant follows
The methodology page lists hard rules that apply to every model, including Jev:
- No truncation. A prompt that does not fit a model's context window is recorded as unsupported, never shortened.
- No label filtering. Every question offers its full option set.
- No prompt tuning. Each model runs with its own published helpers and readout rules.
- No retry on a wrong answer. Transport errors retry up to twice.
- Abstentions stand, and unsupported, refused or errored requests score zero.
Ties matter too. Models less than 0.9 points apart share a rank, because the maintainers say a gap that small is within what a fresh sample of questions could produce 95% of the time. Ranks count groups, not models, so "rank 3" can mean the 6th to 10th rows.
All open models run on one NVIDIA RTX PRO 6000 (96 GB). Jev runs through TypeSafe's hosted API.
Who leads the board right now
On the November 6 data, the top tie groups look like this:
| Tie group | Model | Full score | Built on |
|---|---|---|---|
| 1 | REAX 1.3 | 73.98 | Blend of Gemma and Qwen lineage models |
| 1 | GLiDE 27B (Fastino) | 73.37 | Qwen3.8-27B, full fine tune |
| 2 | KnowLine 31B Gen1.5 | 71.91 | Gemma 4 31B, LoRA |
| 2 | FlexDM LoRA 2.1 | 71.61 | Gemma 4 31B, LoRA |
| 2 | Sieve 31B v3 | 71.10 | Gemma 4 31B, full fine tune |
| 12 | Jev 1.14 | 60.24 | TypeSafe, closed |
Two things stand out in the data. First, every model in the top dozen is built on Gemma 4 31B or Qwen3.8-27B. Second, the leader is itself a combination of other entrants. The REAX 1.3 model card describes two parts that run side by side on one GPU, one from the Gemma line and one from the Qwen line, with their per option probabilities averaged 50/50. Both parts are blends of miner models from a Bittensor subnet with other open entrants such as KnowLine and vLLM-SR d3-ultra. It needs about 115 GB of weights and a 96 GB card, so "open" here does not mean "runs on a laptop".
The lead also moves fast. Jeffan 31B was promoted as number one in late October; on the November 6 data it sits in the third tie group at 69.75, after the November 4 and November 6 updates added more than 20 new models.
Why Jev ranks 55th on its own benchmark
Jev 1.14 scores 60.24 Full and 60.7 public, with 54 models above it. A few points from the methodology help read that result fairly.
The Jev run is a lab run from October 14, 2026, through the TypeSafe API with model id jev-1.14.0, using the same clients, rows and scoring as the earlier Jev 1.13 run. TypeSafe's model docs list Jev 1.14 as the newest release, priced at $0.042 per million input tokens with output free, and a default context of 64k tokens per request and 32k per question. Because the board never truncates, any request beyond those limits would count as unsupported. The board reports full coverage for Jev, so that did not cost it points here.
Calibration is where the gap is clearest. On a sample of about 34,400 requests (the same rows for each model), the board's data shows:
| Model | Accuracy | Mean confidence | Calibration error (ECE) |
|---|---|---|---|
| REAX 1.3 | 81.9% | 83.7% | 0.018 |
| GLiDE 27B | 82.2% | 83.4% | 0.021 |
| Jev 1.14 | 74.3% | 81.7% | 0.080 |
Jev's worst calibrated area is Retrieval & Classification, where its stated confidence averages 86% while its accuracy is 62%. That matters because calibrated confidence is the core of TypeSafe's pitch. Our guide on LLM confidence scores explains what ECE measures and why one number can hide task level gaps.
Keep the caveats in view. Jev's latency on the board (196 ms median) includes the network trip to TypeSafe's servers and is not comparable with the on card figures for open models. The open entrants were often tuned with this exact interface in mind, sometimes on data close to the public suite. And the ranking reflects one community suite, not every workload you might send to a decision model.
Public scores still predict the ranking, mostly
We ranked all 275 models by public score and by Full score and compared the two orderings. The Spearman rank correlation is 0.93, so public results are a decent first guess. The exceptions are the interesting part:
- Base Labs
mammoth-1(27B) is first on the public index (75.34) but 13th on Full score (68.44). - Base Labs
elephant-1(9B) is 7th public and 54th Full. - The biggest drops are small models. Base Labs
joey-1(2B) falls from 58th public to 167th Full, and three 0.8B and 2B models from PKU KCL and StartLux fall by 80 to 103 places.
That pattern fits the maintainers' reason for dropping public scores: small models can look strong on benchmarks they have seen and then lose ground on fresh data. Our guide to LLM leaderboards compared covers the same contamination problem on general model boards.
Jev Decision Index vs Artificial Analysis Decision Index
Searches for "decision index" now hit two different leaderboards. Artificial Analysis runs its own index for decision models, and on November 8, 2026 it released Decision Index v2.
| Item | Jev Decision Index (community) | Artificial Analysis Decision Index v2 |
|---|---|---|
| Who runs it | Community Space on Hugging Face | Artificial Analysis |
| Private share | 100% of the Full score | 45% (up from 25% in v1) |
| Headline metric | Chance corrected accuracy, mapped to a public scale | Decision Score: 50 × (1 + accuracy minus Brier score) |
| Includes chat LLMs | No (decision models and Jev) | Yes (Claude Haiku 5.5, GPT-6 Luna) |
| Current leaders | REAX 1.3, GLiDE 27B | Drex Max v1.1 and GLiDE v1.2 at 73, Mammoth 27B at 72 |
| Cost and speed | Latency on one GPU | Cost per 1M tasks and time per task |
The Artificial Analysis version builds calibration straight into the headline number, adds a 1,500 document private extraction test (AA-Paperclip, 30%) and a consistency test that rewords equivalent questions (AA-Symmetry, 15%). It also reports that Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) 76, ahead of every decision model, but at much higher cost and time per task. Our explainer on the Artificial Analysis Intelligence Index v5 covers how that company builds its other indexes.
Model versions differ between the two boards, so do not line up rows blindly. Drex Max v1 sits in the sixth tie group on the community board at 66.6, while Artificial Analysis tested Drex Max v1.1. A third board, the academic SO-Arena benchmark, uses public items, adds tool and web episodes and reports reliability as its own score.
How to use the board
- Shortlist by tie group, not by row. Anything in the same group is a statistical tie.
- Open the per model page. The Space has a report for each entrant (
index.html?model=<engine>), with area radars, every benchmark, coverage and latency. - Check coverage. Some models accept only up to 8, 16 or 24 options per question, and unanswered requests count as wrong. The methodology lists each model's gaps and reasons; Verdict, for example, left 59% of requests unanswered, mostly by choosing its own abstain option.
- Check the hardware. Most leaders are 27B to 31B models, and REAX 1.3 needs a 96 GB card.
- Run your own labels. Neither board replaces a few hundred labeled examples from your own task. DeepEval can run metrics through Jev directly, and the kit can score any model that speaks the same interface.
FAQ
Is the Jev Decision Index made by TypeSafe?
No. The Space says it is unofficial, community maintained and not affiliated with TypeSafe AI. Jev appears on it because the maintainers ran it through the public API with the same rows and scoring as every other entrant.
Why does Jev score lower than open reproductions?
On the November 6 data, Jev 1.14 has lower accuracy (74.3%) and higher calibration error (0.080) than the leaders on the board's sample, with its worst calibration in retrieval and classification. Many open entrants were tuned for this suite's interface, and the Jev result is a single lab run from October 14.
What is the difference between the Jev Decision Index and the Artificial Analysis Decision Index?
The community board ranks only decision models on 100% private tests mapped to a public scale. Artificial Analysis v2 includes chat models, makes 45% of its index private and uses a Decision Score that rewards calibrated confidence.
Can I run the Decision Index myself?
You can run the public part. The GitHub kit rebuilds the 37 benchmark suite from public sources (about 7 GB of downloads) and scores a model with the board's own scorers. The private tests are run only by the maintainers on submitted models.
How often is the board updated?
There is no fixed schedule. The Space history shows new entrants on October 28, November 3, November 4 and November 6, 2026, often in batches from pull requests. The scoring rules change with each edition, so note the edition when you quote a score.


