Ragas vs LangSmith for RAG Evaluation (2026)

Ragas vs LangSmith is mostly a false choice, because they sit at different layers. Ragas is a free, open source Python library that computes RAG metrics such as faithfulness and context recall; it has no dashboard, no hosted datasets and no production monitoring. LangSmith is a paid platform from LangChain (with a free Developer tier) that stores datasets, runs experiments, keeps traces and scores live traffic, and it accepts any Python function as an evaluator, including a Ragas metric. Use Ragas alone for quick local scoring, LangSmith alone if its built in RAG judges are enough, and both when you want Ragas scores tracked over time. Before you build on Ragas, know that its last release was in January 2026.

We tested the combined setup on Nov 2, 2026: three Ragas metrics wired into the LangSmith SDK as evaluators, run locally on a small test set. The results and two surprises are below.

Ragas vs LangSmith at a glance

What Ragas does well

Ragas gives you a menu of RAG specific metrics with clear definitions. Faithfulness, for example, splits an answer into claims, asks a judge model whether each claim is supported by the retrieved context, and divides supported claims by total claims (Ragas faithfulness docs). It also ships metrics that need no model at all, such as ID based context recall and precision, plus a test set generator for building synthetic questions from your documents.

The strengths are price and portability. Ragas costs nothing beyond the judge calls, runs anywhere Python runs, and needs no account. Our RAG evaluation metrics guide explains each metric and how Ragas names compare with DeepEval and TruLens.

What Ragas does not give you is a place to keep results. You get a score table in memory or a file; comparing last week's run with today's, sharing failures with a teammate or scoring production traffic is your job.

What LangSmith does well

LangSmith is built around the loop of dataset, experiment and trace. You store examples with inputs and reference outputs, run your app against them with evaluate(), and every run becomes an experiment you can compare example by example. LangChain's documentation separates offline evaluation on curated datasets from online evaluation on production runs and threads, where no reference answer exists (LangSmith evaluation concepts).

For RAG specifically, LangSmith's RAG evaluation tutorial defines four LLM judges: correctness (answer against reference), relevance (answer against question), groundedness (answer against retrieved documents) and retrieval relevance (documents against question). Each one returns a structured true or false grade with an explanation. Only correctness needs a reference answer.

On top of that you get trace storage, annotation queues for human review, pairwise comparisons, and online evaluators that run on a filtered, sampled share of live traces. If your app runs on LangGraph, our guide to using LangSmith with LangGraph walks through tracing and graph evals step by step.

Ragas metrics vs LangSmith evaluators, side by side

The practical difference is granularity. A true or false grade tells you how many answers failed; a 0 to 1 faithfulness score tells you how much of each answer was unsupported. Both rely on a judge model, so validate either one against a small human labeled sample first, as our LLM as a judge guide explains.

How to run Ragas inside LangSmith (tested)

LangSmith evaluators are plain functions that receive the run outputs and the example's reference outputs and return a dict with a key and a score. That makes wrapping a Ragas metric short:

from langsmith import evaluate
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import IDBasedContextRecall

recall = IDBasedContextRecall()

def ragas_id_recall(outputs, reference_outputs):
    sample = SingleTurnSample(
        retrieved_context_ids=outputs["doc_ids"],
        reference_context_ids=reference_outputs["doc_ids"],
    )
    return {"key": "ragas_id_context_recall",
            "score": recall.single_turn_score(sample)}

results = evaluate(target, data=examples,
                   evaluators=[ragas_id_recall],
                   upload_results=False)

Our test used a four question support bot set, a toy keyword retriever that returns the top two of five documents, and a generator that simply answers with the top document. We wrapped three Ragas metrics this way: ID based context recall, ID based context precision and NonLLMStringSimilarity. Here is what came back:

Two lessons from the numbers. The recall column caught the real bug: "fast delivery" shares no words with the "Express shipping" document, so the retriever missed it. The string similarity column misled: "Items can be returned within 30 days of purchase" is a correct answer to the first question, yet it scored 0.245 against the reference "You have 30 days to return an item." Use string metrics only for exact formats. For answer quality, use a judge metric and check it, which is what our RAG evaluation through Python walkthrough found too.

Two surprises from the test

Local mode still phoned home. With upload_results=False and no API key, the scores came back locally, but the SDK still tried to send the runs to api.smith.langchain.com and logged a 401 Unauthorized error. Setting LANGSMITH_TRACING=false did not stop the attempt in our run. Treat local mode as a way to try evaluators before you sign up, not as a privacy control. If test data must not leave your network, run Ragas on its own or ask LangChain about self hosting.

Ragas cannot run as a LangSmith online code evaluator. Code evaluators written inside the LangSmith UI may import only the standard library plus numpy, pandas, jsonschema, scipy and scikit-learn, and they have no internet access (online code evaluator docs). To score production traces with Ragas you have to pull runs with the SDK, score them on your own machine and write the scores back as feedback. That works, but it is a job you schedule and maintain.

Older Ragas docs describe the reverse integration: set the LangSmith environment variables and the LLM calls Ragas makes during an evaluation appear as traces, because Ragas is built on LangChain (Ragas LangSmith integration). That page still uses the older LANGCHAIN_ variable names. It helps when you want to debug why a judge gave a strange score.

Is Ragas still maintained?

This is the biggest risk in the comparison, so check it yourself before you commit. On Nov 2, 2026, the latest Ragas release was v0.4.3 from January 13, 2026, and the most recent commit on the main branch of the Ragas GitHub repository was dated February 24, 2026. Before that, releases came every few weeks.

Users have noticed. An issue titled "Is this project actively maintained at this point?" was opened on September 10, 2026 (issue #3004), and when we checked it had no reply from the maintainers. A security advisory, CVE-2026-6587, describes a server side request forgery flaw in the multimodal faithfulness module affecting versions 0.2.3 through 0.4.3, with no patched version listed; GitHub rates it low severity (GitHub advisory). And as our Python walkthrough documented, 0.4.3 fails to import with current langchain-community releases unless you pin it below 0.4.

None of this breaks the metrics themselves; ID based recall gave correct numbers in our test. But it changes the advice. Pin your Ragas version, avoid the multimodal metrics on untrusted inputs, and keep your evaluators thin enough to swap. Wrapping metrics in LangSmith evaluator functions, as above, helps, because the experiment history lives in LangSmith even if you later replace Ragas with DeepEval or your own code.

What each costs

Ragas is free under Apache 2.0. Your costs are the judge model calls, and they add up: in the 0.4.3 source, faithfulness makes two model calls per example, one to split the answer into claims and one to verify them. ID based and string metrics cost nothing.

LangSmith's pricing page lists the Developer plan at 0 dollars with one seat and up to 5,000 base traces a month, and Plus at 39 dollars per seat a month with up to 10,000 base traces, then pay as you go. Base traces are kept 14 days and extended traces 180 days. Evaluation is not free of trace counts: the pricing FAQ says a trace can be an agent, an evaluator or a playground session. LangSmith also sells Tuned Evaluators, metered at 0.015 LSU per evaluation run (one LSU is one dollar), in public beta on Plus and Cloud Enterprise in the US. For key types and limits, see is the LangSmith API free.

Which should you pick?

• Ragas alone if you are one developer checking a retriever, want no account, or must keep data local. Pin the version.
• LangSmith alone if you already trace with LangChain or LangGraph and true or false judges for groundedness and correctness are enough. You get history, review queues and online scoring in one place.
• Both if you want Ragas's finer metrics, especially ID based recall and 0 to 1 faithfulness, tracked as LangSmith experiments. Wrap each metric as an evaluator and run offline; score production traces with a scheduled script.
• Neither if you want an open source platform instead: compare Langfuse vs LangSmith, or see our LLM evaluation frameworks roundup for DeepEval and others.

FAQ

Can I use Ragas with LangSmith?

Yes. Write each Ragas metric as a small function that returns a key and a score, and pass it to LangSmith's evaluate() as an evaluator. We ran three Ragas metrics this way on Nov 2, 2026. Ragas metrics cannot run as online code evaluators inside the LangSmith UI, because those may only import a short list of packages.

Is Ragas better than LangSmith for RAG evaluation?

They do different jobs. Ragas has more RAG specific metrics and finer scores; LangSmith has datasets, experiment history, human review and production monitoring. Many teams use Ragas for the scoring and LangSmith for everything around it.

Is Ragas still maintained in 2026?

Activity has stalled. The last release was 0.4.3 on January 13, 2026, and the last commit on the main branch was February 24, 2026, as of Nov 2, 2026. An open security advisory covers versions up to 0.4.3. It still works if you pin dependencies, but plan for a replacement.

Does LangSmith have built in RAG metrics?

LangSmith's RAG tutorial gives ready made LLM judges for correctness, relevance, groundedness and retrieval relevance, and LangChain's open source openevals package has prebuilt RAG prompts for the same ideas. You can also write any code evaluator yourself.

Is Ragas free?

Yes. Ragas is open source under Apache 2.0. The only cost is the LLM you use as a judge for metrics like faithfulness; metrics based on IDs or strings run on your CPU for free.