The best LLM judge model for most teams in November 2026 is not one model but two: a cheap, well calibrated judge that grades every output, plus a frontier model that only sees the cases the cheap judge is unsure about. A frontier LLM alone is still the most accurate single judge in the published tests, but in the best published test it cost roughly 39 to 77 times more per verdict than a decision model such as TypeSafe's Jev or OpenAI's GPT-6 Luna on the Decisions API. A decision model alone is fast and cheap, yet it trails the frontier model by several points on hard cases. Hightouch's November 4 test found that sending only the uncertain middle of Jev's scores to Claude Opus 5.5 came within 0.2 points of Opus accuracy at under a quarter of the cost. The catch: this only works when the cheap judge's confidence is honest, so you have to measure calibration on your own labels first.
The three kinds of LLM judge model
An LLM judge is a model that grades another model's output against a criterion, as our LLM as a judge guide explains. What changed in the fall of 2026 is the range of models you can put in that seat.
| Judge type | Examples | Output | Strength | Weakness |
|---|---|---|---|---|
| Frontier LLM | Claude Opus 5.5, GPT-6 Sol, Gemini 4 Argon | Text verdict, often with a written reason | Highest accuracy on hard, ambiguous cases | Slowest and most expensive; stated confidence is unreliable |
| Small or open judge | Qwen3.6-35B, Nemotron-3.5-Content-Safety, Prometheus 2 | Text verdict | Self hosted, private, cheaper at volume | Needs prompt tuning; accuracy varies by task |
| Decision model | Jev (TypeSafe), GPT-6 Luna on the Decisions API, Gemini Decision 1 | A probability, choice or score, never free text | Very cheap per call, typed output, a probability on every answer | Cannot explain itself; trails frontier models on the hardest cases |
Decision models are the new entry. TypeSafe launched Jev in early access on September 15, 2026, describing it as a model that gives up text generation and returns "typed probabilistic decisions" (TypeSafe launch post). OpenAI followed with the Decisions API, which is now generally available and supports one model, gpt-6-luna. Both take some input and a list of bounded questions (is this true, which of these options, how severe on this scale) and return numbers instead of prose. For an eval, that maps neatly onto a pass or fail rubric. To compare open decision models with Jev on the same tests, see our Jev Decision Index guide.
Google joined on November 11, 2026 with the Gemini Decisions API and its gemini-decision-1 model. Google says it answers up to 1,000 questions about one text, image, audio, video or PDF input in 150 to 250 milliseconds, with the lowest calibration error in Google's own benchmark suite. The independent tests below predate it; the first outside run we know of is the SO-Arena benchmark, where it scores 61.2.
Open models trained to grade, such as Prometheus 2, sit in the middle group and suit data that cannot leave your servers.
What the November 2026 Hightouch test found
Hightouch, a customer data company, needed to grade identity resolution: deciding whether a merged customer profile really describes one person or wrongly merges two. It built 1,000 synthetic profiles from the pseudopeople simulator (half clean, half wrongly merged) and ran five judges through the same prompt (Hightouch write up, November 4, 2026). The post drew wide discussion on Hacker News on November 6.
| Judge | Accuracy (first test) | Time per profile | Cost per 1,000 profiles |
|---|---|---|---|
| Claude Opus 5.5 | 99.0% | 2.2 s | $3.86 |
| Gemini 3.1 Pro | 98.3% | 4.2 s | $4.48 |
| GPT-6 Sol | 98.3% | 2.1 s | $2.50 |
| GPT-6 Luna, Decisions API (70% cutoff) | 97.3% | 0.17 s | $0.10 |
| Jev (60% cutoff) | 95.7% | 0.15 s | $0.05 |
On the first test, Luna looked almost as good as the frontier models for about one fortieth of the cost. Hightouch then built a harder set made only of tricky cases (typos in birth dates, twins, a junior and a senior with the same name). There, Opus scored 95.1%, Jev alone 89.1% and Luna alone 88.8%. The cheap judges fell about six points behind.
The interesting result came from combining them. Hightouch sent every profile to Jev first and forwarded only the ones where Jev's probability sat between 30% and 90% to Opus. On the first test, that meant 22% of profiles went to Opus, and accuracy reached 98.8% at $0.88 per 1,000. On the harder test, the same rule sent more cases to Opus automatically and scored 95.3% at $2.94 per 1,000, slightly above Opus alone at $5.39. The same trick with Luna reached only 91.6% on the hard set.
Hightouch notes that costs and latency are from its own setup and will vary. It is one company, one task and one synthetic dataset.
What Red Hat's guardrail benchmark found
A second test points the other way. Red Hat's AI safety team compared nine guardrail methods on prompt injection and content safety datasets, including small pretrained classifiers, LLM judges and three decision models (Red Hat Developer, October 2, 2026).
| Method | Type | Prompt injection accuracy | Content safety accuracy |
|---|---|---|---|
| Qwen3.6-35B | LLM judge | 89.31% | 85.47% |
| deberta-v3-base-prompt-injection-v2 | Pretrained classifier | 89.01% | n/a |
| DiffusionGemma via vLLM | Decision model | 87.72% | 85.53% |
| Jev 1.13.0 | Decision model | 86.35% | 86.20% |
| Nemotron-3.5 (custom policy) | Small LLM judge | 84.84% | 85.07% |
Jev led content safety by a small margin, and a 200 million parameter classifier nearly matched the best result on prompt injection (our prompt injection detection library guide compares that model with its successors). Red Hat's conclusion was blunt: in its tests, decision models were not faster, cheaper or more accurate than an LLM judge. Its tests ran from the United Kingdom against US servers, which added network time to Jev's 348 to 360 millisecond median latency.
Red Hat also found that prompts do not transfer. A policy tuned for the open decision model Laya raised Laya's content safety accuracy by 17.83 points but lowered Jev's by 3.67 points. Whatever judge you pick, tune and test its instructions for that judge.
The two tests do not contradict each other. Red Hat compared judges one at a time on binary safety labels. Hightouch found the payoff in combining a cheap judge with an expensive one, which depends on a property Red Hat did not measure: calibration.
Why calibration decides whether a cheap judge works
A cascade works only if the cheap judge knows when it might be wrong. If its mistakes cluster around 50% confidence, you can route that band to a stronger model and trust the rest. If it is wrong at 95% confidence, no threshold saves you.
That is exactly what separated the two cheap judges in the Hightouch test. Most of Jev's errors came with middling confidence. Luna made fewer errors on the easy set, but many of them came with confidence in the 90% to 100% range, so a threshold could not catch them. TypeSafe says Jev is trained with a method it calls Reinforcement Learning for Calibrated Decisions (TypeSafe model docs), but independent checks covered in our LLM confidence score guide found its calibration still varies by task.
The idea is older than either product. The 2024 paper Trust or Escalate by Jaehun Jung, Faeze Brahman and Yejin Choi proposed "Cascaded Selective Evaluation": cheaper judges go first and stronger ones are used only when needed, with a statistical guarantee of agreement with human raters. On a Chatbot Arena subset where GPT-4 almost never reached 80% human agreement, their method guaranteed over 80% agreement with almost 80% coverage while using models as small as Mistral-7B.
How to build a cascaded LLM judge
- Label first. Grade 100 to 200 real outputs by hand for the one failure mode you care about. This is your test set; without it you cannot tune anything.
- Write one narrow question. "Does the answer state a fact not found in the retrieved passages?" works for any judge type. "Rate quality from 1 to 10" works for none.
- Run the cheap judge on the whole set and record its probability for every item, not just the verdict.
- Check calibration. Bucket the probabilities (0 to 10%, 10 to 20% and so on) and compare each bucket's accuracy with its stated confidence. If high confidence buckets contain many errors, that judge is a poor first stage.
- Pick an escalation band. Send items inside the band (Hightouch used 30% to 90% for Jev) to the frontier judge. Widen the band until accuracy on your labels matches the frontier judge alone, then stop.
- Pin versions. TypeSafe's docs warn that the
jev-latestalias moves when a new release ships, so pinjev-1.14.0once you have tuned thresholds. Do the same for the frontier model. - Revalidate on a schedule. Rerun the labeled set whenever a model, prompt or input distribution changes.
If you already use DeepEval, much of this is built in. Its eval modes let a metric run as llm (the default), hybrid (an LLM extracts claims and writes the reason while Jev makes each decision, falling back to the LLM if a Jev call fails) or system_one (Jev only, no LLM key needed). GEval ignores the setting. Our Inspect AI vs DeepEval comparison covers the rest of the framework, and Promptfoo can run the same rubric across several judge models side by side.
What a judgment costs
Prices below are list prices from each vendor's page on November 9, 2026, plus Gemini Decision 1 from November 12. Your token counts decide the real bill.
| Judge | Pricing basis | Source |
|---|---|---|
| Jev 1.14 | $0.042 per million input tokens; output free | TypeSafe model docs |
| GPT-6 Luna on the Decisions API | $0.10 per million uncached input tokens; output not charged | OpenAI Decisions guide |
| Gemini Decision 1 | $0.10 per million input tokens; output not charged; free tier for prototyping in Google AI Studio | Google launch post |
| Frontier LLM judges | Standard input and output token rates for each model | Each provider's pricing page |
Because decision models charge nothing for output, a long written reason is where frontier judges spend most of your money. In DeepEval's hybrid mode, the LLM still writes reasons, so you save on decisions but not on explanations.
When to use each judge model
- Use a frontier LLM alone for small offline eval runs (a few hundred items), for hard or subjective rubrics, and whenever you need a written reason a human will read.
- Use a decision model alone for high volume, narrow binary checks where a few points of accuracy matter less than speed, such as tagging traces in Langfuse or a first pass on production logs. Validate it on labels first.
- Use a cascade when you grade thousands of items a day and need frontier accuracy, and your cheap judge's errors cluster at middling confidence.
- Use a small pretrained classifier for well defined safety risks with lots of labeled data, as Red Hat's prompt injection result shows. Our prompt injection attacks guide covers that threat.
Limits of these results
Every number here comes from a vendor or a single team. Hightouch sells customer data software, not models, but its test is one task on synthetic records. Red Hat ships the classifiers that did well in its own test and says so. TypeSafe's own benchmark uses the average of GPT-6 Astra and Fable 5.1 as the reference answer, which it notes biases results toward those labs. None of these tests covers RAG faithfulness, where our LLM evaluation metrics guide is a better starting point, or agent runs, where an agent judge with tools may fit better. We have not run Jev or the Decisions API ourselves for this guide.
FAQ
What is the best LLM judge model?
For a single judge, a frontier model such as Claude Opus 5.5 was the most accurate in Hightouch's November 2026 test (99.0% on the first set, 95.1% on the harder one). For cost and accuracy together, a cascade of a calibrated decision model plus a frontier model did best.
Are decision models better than LLMs as judges?
Not on their own. Red Hat found Jev did not reliably beat an LLM judge or a small classifier on guardrail tasks, and Hightouch found Jev and Luna about six points behind Opus on hard cases. Their advantage is speed, price and a probability on every answer.
What is the difference between Jev and the OpenAI Decisions API?
Both return typed answers with probabilities instead of text. Jev is TypeSafe's model at $0.042 per million input tokens. The Decisions API runs GPT-6 Luna at $0.10 per million uncached input tokens. In Hightouch's test, Luna was more accurate alone on easy cases but more overconfident, which made it worse in a cascade.
Can I use Jev as a judge in DeepEval?
Yes. Set deepeval set-eval-mode hybrid or system_one, or pass eval_mode to a metric, and add a TypeSafe API key. JevEval covers custom metrics decided by Jev.
How much does an LLM judge cost per evaluation?
It depends on token counts. In Hightouch's setup, grading 1,000 profiles cost $3.86 with Claude Opus 5.5, $0.10 with GPT-6 Luna on the Decisions API, $0.05 with Jev and $0.88 with the Jev plus Opus cascade.
Should the judge model be different from the model being judged?
Not necessarily. For narrow pass or fail checks, the same model can work if it agrees with your human labels. Switch judges only when agreement is poor.


