RAG Evaluation Through Python: A Tested Walkthrough

RAG evaluation through Python works best in two layers. First, score retrieval with plain Python: for each test question, compare the document IDs your retriever returned with the IDs a person marked as relevant, and compute hit rate, recall at k, precision at k and mean reciprocal rank (MRR). That needs no API key and about 20 lines of code. Then score the answers, using an LLM judge (for example Ragas Faithfulness) for anything that needs judgment, because string metrics such as ROUGE cannot tell a right answer from a wrong one.

We ran exactly that on Oct 30, 2026: a six question support bot test set, a toy keyword retriever, hand written metric functions and Ragas 0.4.3 as a cross check. The numbers below come from that run, including two surprises: the current Ragas release fails to import with the newest langchain-community, and the documented ROUGE metric gave a wrong answer a far higher score than a correct one. If you want the theory behind each metric first, read our RAG evaluation metrics guide; this page is the hands on version.

What you need before writing any evaluation code

Every RAG evaluation in Python starts with a small labeled dataset, not a library. Each row needs three things:

• The question, ideally copied from real user traffic.
• The relevant document IDs: which chunks in your index actually contain the answer. A person decides this once.
• A reference answer: one or two sentences a domain expert would accept.

Our test set used an eight chunk knowledge base (return policy, shipping, account and warranty snippets) and six questions. Here is one row as a Python dict:

{"q": "Can I get a refund on a gift?",
 "relevant": ["returns-3", "returns-2"],
 "reference": "Gift items get store credit only, not a refund."}

Storing chunk IDs is the most useful habit in this whole guide. With IDs, retrieval scoring is exact set arithmetic. Without them you are stuck comparing text, which is slower and fuzzier, as the next sections show.

Step 1: score retrieval in plain Python

Retrieval metrics only compare two lists, so you do not need a framework for them. These four functions are the whole toolkit:

def hit_rate(ret, rel):        # did any relevant chunk come back?
    return 1.0 if set(ret) & set(rel) else 0.0

def recall_at_k(ret, rel):     # share of relevant chunks retrieved
    return len(set(ret) & set(rel)) / len(rel)

def precision_at_k(ret, rel):  # share of retrieved chunks that are relevant
    return len(set(ret) & set(rel)) / len(ret)

def reciprocal_rank(ret, rel): # 1 / position of first relevant chunk
    for i, d in enumerate(ret, 1):
        if d in rel:
            return 1.0 / i
    return 0.0

Run your retriever on each question with k set to the number of chunks you actually pass to the model (we used 3), apply the functions, and average across the set. Our keyword retriever produced this:

The averages look decent, but the table tells the real story. The one miss is a vocabulary problem: the user said "fee" and "fast delivery", the chunk says "costs" and "express shipping", and a keyword retriever has no way to connect them. That points straight at the fix (embeddings or a hybrid retriever), which an average alone would never tell you.

Precision@3 of 0.33 is normal here. Most questions have one relevant chunk, so even a perfect retriever that returns three chunks scores 0.33. Read precision against the best possible value for your set, not against 1.0.

Step 2: cross check with Ragas in Python

Ragas is a widely used open source Python library for RAG evaluation, and it ships the same retrieval metrics. The current release on PyPI is 0.4.3, uploaded on January 13, 2026 (PyPI).

The install problem you will probably hit

A fresh pip install ragas on Oct 30 pulled in langchain-community 0.4.2 (released May 22, 2026), and import ragas failed immediately:

ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai'

Ragas 0.4.3 imports a Vertex AI class from a path that newer langchain-community releases removed. It is a known bug, reported in issue #2745 on May 25, 2026 and still open when we checked; a proposed fix in pull request #2769 was also unmerged. What worked for us was pinning the older community package, which that pull request also lists as the version the project locks:

pip install ragas "langchain-community<0.4" rapidfuzz rouge_score

rapidfuzz and rouge_score are optional extras, but the string based metrics below raise an ImportError without them.

The Ragas retrieval metrics

Ragas has two ways to score retrieval without an LLM. IDBasedContextRecall and IDBasedContextPrecision compare chunk IDs, exactly like our functions. NonLLMContextRecall compares chunk text instead: it calls a retrieved chunk relevant when its string similarity to a reference chunk passes a threshold, 0.5 by default in the 0.4.3 source (context recall docs).

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import IDBasedContextRecall, IDBasedContextPrecision

sample = SingleTurnSample(
    retrieved_context_ids=["returns-2", "returns-3", "account-1"],
    reference_context_ids=["returns-3", "returns-2"],
)
recall = await IDBasedContextRecall().single_turn_ascore(sample)       # 1.0
precision = await IDBasedContextPrecision().single_turn_ascore(sample) # 0.667

Across all six questions Ragas matched our hand written numbers exactly: ID recall 0.833, ID precision 0.333 and text based recall 0.833. That is the point of the cross check. When two independent implementations agree, you can trust your pipeline and stop worrying about the metric code.

Two quirks in 0.4.3. These classes live on the older ragas.metrics path, which prints a deprecation warning telling you to import from ragas.metrics.collections instead. But in 0.4.3 that new module has no IDBasedContextRecall (we got an ImportError), so keep the old import for now. The docs say the legacy API will be removed in version 1.0, so pin your Ragas version in requirements.

Step 3: score the answers, and why string metrics mislead

Retrieval scores say whether the evidence was available. You still need to know whether the answer was right. To isolate that, our toy generator simply returned the top chunk as the answer, and we scored it against the reference with Ragas RougeScore (ROUGE L F measure, the default).

The results show the problem:

• Correct answer, different words. "Items can be returned within 30 days of purchase" against the reference "You have 30 days to return an item" scored 0.24.
• Wrong answer. For the gift question, the top chunk was "You need the original receipt to get a refund", which misses the store credit rule entirely. It scored 0.33, higher than the correct answer above.
• Ragas's own example. Rerunning the docs example, "The Eiffel Tower is located in India" against "...located in Paris" scored 0.857 on ROUGE and 0.892 on NonLLMStringSimilarity (traditional metrics docs). We got the same numbers.

String overlap rewards copying the reference's words, not being correct. Use these metrics only where wording is fixed: ExactMatch for IDs, codes or tool arguments, and StringPresence for a must include fact (our check that "30 days" appears in the answer returned 1.0). For anything else, you need a metric that reads meaning.

Step 4: add LLM judged metrics for faithfulness and correctness

This is where most of the value, and all of the cost, sits. Ragas Faithfulness breaks the answer into claims and asks a judge model whether each claim is supported by the retrieved chunks, then reports supported claims divided by total claims (faithfulness docs). The current collections API looks like this:

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
scorer = Faithfulness(llm=llm)
result = await scorer.ascore(
    user_input="Can I get a refund on a gift?",
    response=answer,
    retrieved_contexts=chunks,
)
print(result.value)

We did not run this step (no judge API key on the test machine), so we have no judged scores to report. Some practical points from the docs and from how these metrics work:

• Every judged score costs one or more model calls per row, so run your cheap ID metrics on every commit and the judged ones on a schedule or before release.
• Judged scores vary between runs and between judge models. Fix the judge model and version and record it with every result. Our LLM as a judge guide covers how to check a judge against human labels.
• Faithfulness alone cannot catch our gift failure. "You need the original receipt to get a refund" is fully supported by its chunk, so it would score as faithful while still being wrong for a gift. You also need a reference based check of correctness or completeness; see what is faithfulness in RAG evaluation and answer completeness for how the two halves fit together.
• Ragas also documents FaithfulnesswithHHEM, which swaps the judge LLM for Vectara's small open HHEM classifier for the claim check step and can run on CPU.

DeepEval offers the same judged metric families as pytest style tests if you would rather run evals inside your test suite; our Inspect AI vs DeepEval comparison explains that setup.

RAG evaluation techniques compared

Building a dataset for RAG evaluation

Six questions were enough to show how the code behaves, not to measure a system. With six rows, one question flips any average by about 17 points, which is far too coarse to compare two retrievers. Our RAG evaluation metrics guide suggests starting with 50 to 100 real questions.

A few rules that made our small set useful and scale up well:

1. Include questions that use different words from your documents. Our only retrieval failure came from the one question phrased the way a customer talks.
2. Include questions that need two chunks. The gift question needed both the gift rule and the receipt rule, and it was the only case where precision rose above 0.33.
3. Label chunk IDs, not just answers. Chunk IDs make retrieval scoring exact and keep working when you change prompts or models. Relabel if you change chunking, because IDs shift.
4. Keep the set in version control next to the eval script, and log every run's scores with the commit hash. An observability tool such as Langfuse can store datasets and scores if you want a UI.

Common mistakes in Python RAG evals

• Reporting one blended score. Keep retrieval and answer metrics separate so you know which half broke.
• Trusting averages. Our 0.83 hit rate hid a systematic vocabulary gap. Always print the failing rows.
• Using ROUGE as a quality score. It ranked a wrong answer above a right one in our run.
• Scoring with k different from production. If the app sends five chunks to the model, measure recall@5.
• Leaving versions unpinned. Both bugs we hit came from dependency drift.

FAQ

Can you do RAG evaluation in Python without an API key?

Yes, for retrieval and for exact checks. Hit rate, recall at k, precision at k and MRR need only your labeled chunk IDs and a few lines of Python, and Ragas's ID based and string metrics run locally on CPU. Judging whether an answer is faithful or correct needs a model, either a hosted LLM or a local classifier such as HHEM.

Which Python library is best for RAG evaluation?

For RAG specific metrics, Ragas is the usual starting point, and its ID based metrics matched our hand written code exactly. DeepEval suits teams that want evals as pytest tests in CI. For retrieval scoring alone, plain Python is enough and has no dependencies to break.

Why does import ragas fail with a vertexai error?

Ragas 0.4.3 imports ChatVertexAI from a langchain_community path that newer releases removed. Installing langchain-community<0.4 fixed it for us on Oct 30, 2026. The bug is tracked in Ragas issue #2745, which was still open that day.

What is the basic purpose of RAG evaluation?

To find out which half of the pipeline fails and how often: whether the retriever brings back the evidence, and whether the generator turns that evidence into a correct, supported answer. Separate scores for each half tell you whether to fix search or fix the prompt and model.

What is a good MRR for a RAG retriever?

There is no universal cutoff, because it depends on how many chunks you pass and how hard your questions are. Compare MRR between versions of your own retriever on the same labeled set. An MRR close to your hit rate, as in our run, means relevant chunks usually come first when they come back at all.