LLM Confidence Scores: Can You Trust Them?

An LLM confidence score is a number that is supposed to say how likely a model's answer is to be correct. You can get one four ways: ask the model to state it, read the token probabilities, sample the same question several times and count agreement, or compute it yourself from signals you log. None of them is trustworthy until you grade it against real outcomes on your own task, because a score that sounds precise can be badly calibrated.

The question is back in the spotlight this week. On October 20, 2026, Amrut Joshi published Asking LLM for confidence, which reached the Hacker News front page as "Using LLM for confidence scoring is a bad idea" (discussion, 113 points and 61 comments on the morning of October 22). Commenters pushed back with data, including the ConfidenceBench paper, which found that some frontier models state confidence about as well as a careful human. Both sides are partly right, and the research explains why.

What an LLM confidence score actually is

There is no gauge inside a chat model that reads out certainty. Every confidence score is produced by one of these methods, and each measures something different.

Joshi's argument targets the first row. When you ask "how confident are you?", the reply "0.85" is written token by token like any other text. In his words, it is a plausible continuation of the conversation, not a measurement. He lists three symptoms you can check yourself: ask again and the number changes, change the wording and it changes, and text the model already wrote ("this is clearly right") pulls the number up.

What the research says about verbalized confidence

The evidence is more mixed than "never ask". Four studies are worth knowing.

Pretrained models can be well calibrated in the right format. Anthropic's Kadavath et al. (2022) showed that larger models give well calibrated probabilities on multiple choice and true or false questions when the options are shown with letter labels. Replacing an option with "none of the above" hurt both accuracy and calibration. Their RLHF tuned models looked very miscalibrated, but a single temperature adjustment (T = 2.5) largely fixed that on several evaluations.

For tuned chat models, asking can beat reading probabilities. Tian et al. (2023) tested ChatGPT, GPT-4 and Claude on TriviaQA, SciQ and TruthfulQA. Verbalized confidence was usually better calibrated than the models' own conditional probabilities, often cutting expected calibration error by a relative 50 percent. Asking the model for several candidate answers, each with a probability, improved calibration further. Chain of thought prompting did not.

Plain verbalized scores lean high. Xiong et al. (2023) found that models asked for a confidence mostly answer between 80 and 100 percent, usually in multiples of 5, much like people do. Calibration improved as models got more capable but stayed far from ideal. Sampling five answers and measuring agreement helped a lot on math: for GPT-3.5 on GSM8K, the ability to separate right from wrong answers (AUROC) rose from 54.8 percent to 92.7 percent.

2026 frontier models vary widely. ConfidenceBench asked 15 frontier models 200 private four option questions across spatial reasoning, precise math, word lookup and unknowable facts, three runs each, and scored the stated probabilities with the Brier score (lower is better). A model that guesses at random and says 25 percent every time scores 0.1875.

Five of the 15 models did worse than the random baseline. The most accurate model was not the best calibrated one, newer was not always better (Gemini 3.1 Flash-Lite was the worst calibrated model in the study), and within GPT-5 higher reasoning effort tended to help. The authors note the limits: 200 English multiple choice questions, a private set, and stated rather than logit confidence.

So what does that mean for you? A good model can produce useful stated confidence on a narrow, well formatted task. A different model, a different prompt or a different task can break it. That is the core of Joshi's point: you only know which case you are in after you grade the scores.

How to measure LLM confidence calibration

Calibration means that among all answers given 70 percent confidence, about 70 percent turn out correct. You check it on a labeled set, never on a single answer. Use three numbers together.

• Expected calibration error (ECE): sort answers into confidence bins (10 is common), then average the gap between stated confidence and actual accuracy in each bin. Easy to read, but the result shifts with the number of bins.
• Brier score: the mean squared gap between each probability and the outcome (1 for correct, 0 for wrong). It is a proper scoring rule, so it rewards honest probabilities, and it needs no bins.
• AUROC: how well the score ranks correct answers above wrong ones. This is the number that matters if you plan to route low confidence answers to a person.

Plot a reliability diagram too: stated confidence on one axis, observed accuracy on the other, with the diagonal as perfect calibration.

Watch for one trap. Xiong et al. report that GPT-4 with chain of thought scored 93.6 percent on GSM8K and reached a low ECE of 0.064 by giving every answer 100 percent confidence. The ECE looked excellent, yet the score could not tell a right answer from a wrong one. Always pair a calibration number with a ranking number such as AUROC.

Calibration also appears in public benchmarks. Humanity's Last Exam reports calibration error next to accuracy, and Stanford's HELM treats calibration as one of its core metrics. Our guide on how to evaluate LLM model performance covers sample sizes and error bars for these comparisons.

How to build an LLM confidence score you can trust

This is the practical recipe, combining the research above with Joshi's forecaster approach.

1. Label a sample from your real traffic. A few hundred examples with known correct answers is a reasonable start. Without labels you cannot check any score.
2. Pick your raw signal. If you have logprob access and the task is a fixed choice, read the answer token probability; OpenAI's API specification lets you request logprobs with top_logprobs set from 0 to 20 per position. If not, use a verbalized score with an LLM confidence score prompt that asks for several candidate answers, each with a probability. For high stakes calls, add sampling agreement across 5 or more runs.
3. Add signals you can observe. Joshi's support agent example: did retrieval find a matching article, has the agent answered this type of question before, and do two runs agree? These do not depend on how the model feels.
4. Grade and recalibrate. Compute ECE, Brier and AUROC on held out labels. If the scores run high, fit a temperature or map score bins to observed accuracy. This is the step most teams skip.
5. Set the threshold from measured precision. Choose the confidence level at which accepted answers meet your target accuracy on your labels, then send everything below it to review. Do not pick 0.8 because it sounds safe.
6. Log every decision and outcome. Store the score, the signals and whether the answer was later approved or corrected in a tracing tool such as Langfuse or Braintrust, and rebuild the reliability table each week. Recheck whenever the model, prompt or traffic changes.

The same checks apply to LLM as a judge scores, which are confidence numbers about someone else's output, and to the uncertainty signals described in our LLM hallucination detection guide.

What about models sold as calibrated?

Some vendors now sell models built for scored decisions rather than chat. TypeSafe's Jev, for example, returns typed choices with probabilities instead of text, and its model documentation says it is trained for calibrated decisions. The same page advises pinning a version ID if you tuned thresholds, because the jev-latest alias moves when a new release ships (Jev 1.14 became available on October 14, 2026).

Independent tests still found task dependent gaps. A Verne Systems test on 1,232 Banking77 support messages found answers at confidence 1.0 were right 97.5 percent of the time, while answers stated between 0.6 and 0.7 were right 44 percent of the time, with an overall ECE of 0.086. One temperature fitted on 50 labels cut the median held out error from 0.092 to 0.059. A September 2026 calibration audit preprint found the error changed by 0.071 between two ways of posing the same question. The lesson is the same as for chat models: a calibration claim from a vendor is a starting point, and your own labeled check decides.

Common mistakes

• Trusting a single stated number without a labeled test set.
• Reporting ECE alone, which can look perfect while the score ranks nothing.
• Asking for 0 to 100 on a vague question. Narrow, well scoped questions with fixed options calibrate better, as both Kadavath et al. and the Hacker News commenters found.
• Keeping thresholds after a model upgrade. Calibration belongs to a model version, a prompt and a task.
• Treating confidence as proof. A calibrated 90 percent still means about one answer in ten is wrong.

For the wider set of metrics that sit next to calibration, see LLM evaluation metrics.

FAQ

Can you ask an LLM for a confidence score?

Yes, and for strong models on narrow tasks the stated number can be useful. Tian et al. found verbalized confidence from ChatGPT, GPT-4 and Claude was often better calibrated than their token probabilities. But stated scores tend to cluster between 80 and 100 percent and shift with wording, so grade them on labeled examples before you rely on them.

What is a good Brier score for LLM confidence?

It depends on the task. On ConfidenceBench's four option questions, random guessing at 25 percent scores 0.1875, the best models scored 0.103 and a human tester 0.105. Compare models on the same question set rather than against a universal cutoff.

Are logprobs better than verbalized confidence?

Not always. Pretrained models give well calibrated token probabilities on lettered multiple choice questions, but RLHF tuning tends to make them overconfident. Xiong et al. found white box methods, which read the model's probabilities, only slightly ahead of black box ones, with a narrow gap (0.522 to 0.605 AUROC in one comparison). Test both if your API exposes logprobs.

What confidence threshold should I use for an LLM?

Set it from your own data. Pick the score at which accepted answers reach your target accuracy on a labeled sample, route the rest to review, and recheck after every model or prompt change.

Does chain of thought improve LLM confidence?

Tian et al. found chain of thought did not improve verbalized calibration. Xiong et al. found it raised accuracy, which narrowed the gap to stated confidence, but did little to help the score separate right answers from wrong ones.