What is faithfulness in RAG evaluation? It is the measure of whether a generated answer sticks to the documents your retriever handed the model. A faithful answer makes only claims that can be supported by that retrieved context; an unfaithful one adds, changes or contradicts facts. Most tools score it from zero to one as the share of the answer's claims that the context supports, so 0.5 means half of what the model said has no backing in the passages it was given.
Faithfulness is a statement about the answer and its sources, not about the world. A faithful answer can still be wrong if retrieval pulled a wrong or outdated page, and a true answer can score as unfaithful if the model got it from memory instead of the context. This guide explains where the idea comes from, how the main libraries compute it (they do not all mean the same thing), how to read a score, and the mistakes that make faithfulness numbers look better or worse than they are. For the full set of RAG metrics side by side, see our overview of RAG evaluation metrics.
Faithfulness in one example
The Ragas documentation uses a simple case. The question is where and when Einstein was born. The retrieved context says he was born on 14 March 1879 and was German born.
- High faithfulness answer: "Einstein was born in Germany on 14th March 1879." Both claims appear in the context. Score: 1.0.
- Low faithfulness answer: "Einstein was born in Germany on 20th March 1879." The scorer splits this into two statements. The first (born in Germany) is supported; the second (20 March) is not. Score: 1 out of 2, or 0.5.
That is the whole recipe in most tools: break the answer into short claims, check each claim against the retrieved text, and divide supported claims by total claims.
Where the term comes from
The word was borrowed from summarization research before RAG became common. In a 2020 Google Research paper, On Faithfulness and Factuality in Abstractive Summarization, Maynez and colleagues separated two questions: is a summary faithful to its source document, and is it factual about the world? Their human study of news summaries found hallucinated content in more than 70 percent of single sentence summaries, and they showed that some unfaithful content was still true.
The same paper found that word overlap metrics were poor guides. ROUGE and BERTScore correlated with human faithfulness judgments at around 0.16 to 0.20 (Spearman), while a textual entailment score reached 0.431. That result explains why modern faithfulness metrics check whether the source entails each claim rather than counting shared words.
The 2023 Ragas paper carried the idea into RAG. It defined an answer as faithful "if the claims that are made in the answer can be inferred from the context," and made it one of three reference free metrics alongside answer relevance and context relevance.
How faithfulness is calculated
The claim and verify method
The Ragas paper describes two LLM calls. First, a model rewrites the answer as a set of short, focused statements. Second, the model checks each statement against the retrieved context and gives a yes or no verdict with a brief explanation. The score is the number of supported statements divided by the total.
The authors tested this on WikiEval, a set they built from 50 Wikipedia pages about events from 2022 onward, with two human annotators judging which of two answers was more faithful. The claim and verify method agreed with the humans 95 percent of the time. Asking the model for a single zero to ten faithfulness score agreed 72 percent of the time, and asking it to pick the better answer agreed 54 percent of the time. Breaking the answer into claims is what makes the metric work.
Faithfulness formula at a glance
Faithfulness = supported claims ÷ total claims in the answer
It needs three inputs: the question, the retrieved contexts and the response. It does not need a reference answer, which makes it the standard metric for reference free RAG evaluation on live traffic.
Same name, different definitions
This is the part most explainers skip. Libraries use the word faithfulness for checks that are close but not identical, so scores from different tools are not interchangeable.
| Tool | Name | Claim counts as faithful when | Model used |
|---|---|---|---|
| Ragas | Faithfulness | It can be inferred from the retrieved context | LLM judge |
| Ragas | FaithfulnesswithHHEM | Same, with the check done by a classifier | Vectara HHEM-2.1-Open |
| DeepEval | FaithfulnessMetric | It does not contradict the retrieved context | LLM judge (other eval modes available) |
| TruLens | Groundedness | Evidence for it is found in the retrieved context | LLM judge feedback function |
The key difference is between "supported" and "not contradicted." DeepEval's faithfulness docs say a claim "is considered truthful if it does not contradict any facts presented in the retrieval_context." Under that rule, a claim the context never mentions can pass. Ragas asks whether the claim can be inferred from the context, so the same unmentioned claim fails. DeepEval offers a penalize_ambiguous_claims option that stops borderline claims from counting as faithful, which moves it closer to the stricter reading. Its default pass threshold is 0.5, and strict_mode turns the score into a plain 1 or 0.
DeepEval also draws a line between its faithfulness metric, which grades the output against the retrieved context, and its separate hallucination metric, which compares output against a known correct context you supply. Pick the one that matches the data you actually have.
TruLens calls the idea groundedness and makes it one leg of its RAG triad, next to context relevance and answer relevance. Its method is the same in spirit: separate the response into claims and look for evidence for each one in the retrieved context.
Faithfulness without an LLM judge
You do not have to pay an LLM for every check. Vectara's HHEM-2.1-Open is a small open model based on T5 that takes a premise and a hypothesis and returns a score from zero (not supported) to one (fully supported). Vectara reports it runs in under 600 MB of RAM and takes about 1.5 seconds for a 2,000 token input on a modern x86 CPU. On the RAGTruth question answering subset, its model card lists a balanced accuracy of 74.28 percent against 74.11 percent for zero shot GPT-4. Ragas can plug it into the verification step, and the same family of models powers the Vectara hallucination leaderboard.
The model card also gives the clearest example of why faithfulness differs from truth. With the premise "The capital of France is Berlin," the hypothesis "The capital of France is Paris" is scored as hallucinated, because it is not supported by the text provided, even though it is correct in the real world.
How to read a faithfulness score
A single number hides a lot, so read it with these points in mind.
- Low faithfulness with good retrieval means a generation problem. If context recall is high but faithfulness is low, the right passages were there and the model ignored or embellished them. Tighten grounding instructions, ask for citations, or shorten the context.
- High faithfulness with bad retrieval means a confident wrong answer. The model faithfully repeated a stale or irrelevant passage. Check context precision and recall before celebrating.
- Faithfulness says nothing about completeness. "I could not find that in the documents" contains no unsupported claim. Pair faithfulness with answer relevancy so evasive answers do not look perfect.
- Short answers are noisy. An answer with two claims can only score 0, 0.5 or 1. Look at averages across many questions, and read the low scorers.
- Compare like with like. A 0.8 from Ragas and a 0.8 from DeepEval are not the same measurement. Keep the library, judge model and settings fixed when you compare runs.
Common mistakes that skew the metric
Changing the judge model. Claim extraction and verification are both model calls. Swapping the judge, or its prompt, can move scores even when your app did not change. Pin the judge and record its version with every run, the way our LLM as a judge guide recommends.
Passing the wrong context. The metric only sees what you give it. If you log the top 20 chunks but the prompt only included the top 5 after reranking, you will score against text the model never saw. Capture the final prompt context from your traces in a tool such as Langfuse.
Treating general knowledge as an error to fix. Sometimes a model adds a true, harmless fact from memory. Faithfulness will mark it down. Decide as a product question whether your app should ever do that. In regulated or support settings the answer is usually no, which is exactly why the metric is strict.
Never checking the checker. Label 50 to 100 answers by hand as faithful or not, then compare with the metric. If they disagree often, adjust the prompt template (DeepEval and Ragas both let you override it) or switch the verification step to a classifier.
When to use faithfulness, and what to pair it with
Use faithfulness whenever answers must stay inside a known body of text: support bots, policy and legal search, internal knowledge bases and document summaries. Run it in two places: on a fixed test set in CI, and on a sample of production traces, since it needs no reference answers. Our guide on how to evaluate LLM applications covers how to wire both loops.
Pair it with context recall and precision for retrieval, answer relevancy for usefulness, and factual correctness against reference answers where you have them. For catching unsupported claims outside RAG, such as open questions with no source documents, see LLM hallucination detection, and for where faithfulness sits among BLEU, ROUGE and judge scores, see LLM evaluation metrics.
FAQ
What is faithfulness in RAG evaluation, in one sentence?
It is the share of claims in a generated answer that the retrieved context supports, scored from zero to one, so it tells you whether the model stuck to its sources.
Is faithfulness the same as groundedness?
In practice, yes. TruLens uses the name groundedness for the same idea: split the response into claims and look for evidence for each one in the retrieved context. The exact rules and prompts differ by library, so the numbers are not directly comparable.
What is the difference between faithfulness and answer relevancy?
Faithfulness checks that the answer is supported by the context. Answer relevancy checks that the answer addresses the question. An answer can be fully faithful and useless, or on topic and full of unsupported claims, so you need both.
What is a good faithfulness score?
There is no universal cutoff. DeepEval's default pass threshold is 0.5, which is a starting point rather than a target. Set your own bar by labeling a sample of answers by hand and choosing the score that separates the ones you would ship from the ones you would not.
What is MRR in RAG evaluation?
MRR, or mean reciprocal rank, is a retrieval metric, not a generation one. For each question you take one divided by the rank of the first relevant chunk (1 if it is first, 0.5 if second), then average across questions. It measures how high the first useful passage lands, while faithfulness measures whether the answer used the passages correctly.
Can you measure faithfulness without an LLM?
Yes. Natural language inference style classifiers such as Vectara's HHEM-2.1-Open score whether a source supports a statement, run on a CPU, and can replace the LLM verification step in Ragas. They still need claim splitting or short answers to work well.
