RAG Evaluation: How to Measure a RAG Pipeline With Ragas

A retrieval-augmented generation (RAG) pipeline for a chatbot passes every smoke test in the notebook. Then support tickets start coming in about wrong or odd answers, and nothing in the smoke test logs shows a red flag.

This failure is hard to catch because it is invisible without metrics. A RAG pipeline has 2 places to go wrong: the retriever can fetch the wrong chunks, or the generator can ignore good chunks and invent facts. A single end-to-end accuracy number cannot tell you which one happened, so it cannot tell you what to fix.

In this tutorial, I will build a complete RAG evaluation pipeline in Python using Ragas 0.4.3: a hand-built evaluation dataset, the 4 core metrics (faithfulness, answer relevancy, context precision, and context recall), a continuous integration (CI) quality gate that fails the build when scores drop, a synthetic test set generator, and a JSON Lines score tracker.

If you want more background first, see our guide to what RAG is, our tutorial on using a large language model (LLM) as a judge for evaluating RAG pipelines, and our overview of advanced RAG techniques.

You need Python 3.9 or newer (I use 3.12), uv or pip, an API key for an OpenAI-compatible judge model, and a rough idea of how a RAG pipeline works. If you want that grounding first, I recommend our Retrieval-Augmented Generation with LangChain course, which builds the kind of pipeline we will be scoring.

RAG Evaluation in a Nutshell

Text-overlap metrics like BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are the wrong ruler for RAG. Ragas instead uses a judge LLM to score each answer on 4 axes: faithfulness and answer relevancy diagnose the generator, while context precision and context recall diagnose the retriever.

Put those 4 numbers behind a threshold in CI and log them per commit, and you have a regression alarm instead of a one-off audit.

One warning before we start. The Ragas API has changed twice in 2 years: v0.2 replaced dictionary datasets with SingleTurnSample and EvaluationDataset, and v0.4 made the collections metrics (introduced in v0.3.9) the default and marked the old evaluate() function deprecated.

Most tutorials online still show v0.1 or v0.2 code. Everything below runs on 0.4.3, which is the current release as of October 2026.

Why Does RAG Need Its Own Evaluation Metrics?

RAG needs its own metrics because classic text metrics only count how many words an answer shares with a reference answer, and shared wording is not what breaks in a RAG system.

BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) were built for machine translation and summarization, where a good output should use much the same words as the reference. A RAG answer that says “you have 2 weeks” instead of “14 days” scores poorly on both, while a fluent answer that invents a refund policy can score well if it happens to reuse the reference’s words.

The deeper problem is that a RAG pipeline can fail in 2 separate places: the retriever can pick the wrong information, or the generator can ignore good information and invent facts. When an answer is wrong, you need to know which of these happened:

• The right chunk was never retrieved.
• The right chunk was retrieved but ranked too low to be passed to the model.
• The right chunk was passed along and the model ignored it.

The 2 failure modes of a RAG pipeline: context precision and recall diagnose the retriever, while faithfulness and answer relevancy diagnose the generator. Image by author.

Our What Is Retrieval Augmented Generation (RAG)? A Complete Guide walks through the pipeline stages where those failures originate.

Ragas is an open-source Python framework that gives you a number for each part of the pipeline. It is not the only option. The 2 main alternatives are DeepEval, which wraps evaluation in pytest-style test cases, and Evidently, which adds LLM checks as extra columns on a dataset and feeds them into monitoring reports.

I picked Ragas for this tutorial because its 4 core metrics split cleanly into retriever scores and generator scores, and because its test set generator ships in the same open-source package.

If your team already lives in pytest, DeepEval’s assert_test() will feel more natural. If what you want is a dashboard of live traffic quality, Evidently’s reports get you there faster.

How Do I Set Up Ragas and a Judge Model?

Setting up Ragas means installing the package, pinning one dependency, and choosing a judge LLM that every metric will share. I will use uv because it locks exact package versions for you, but pip install with the same versions works too.

uv init rag-eval --python 3.12 && cd rag-eval
uv add "ragas==0.4.3" "langchain-community<0.4" openai pandas matplotlib rapidfuzz

I am using Python 3.12 here. Ragas 0.4.3 declares support for Python 3.9 and newer.

The package rapidfuzz is needed by the synthetic test set generator later in the article. Ragas does not declare it as a dependency, so I install it up front.

The langchain-community<0.4 pin is not optional as of this writing. Ragas 0.4.3 imports ChatVertexAI from a module path that langchain-community 0.4 removed, so a fresh install without the pin fails when you import ragas.

Next, the judge. Every LLM-based metric in Ragas makes 1 or more calls to a judge model, so you build it once and pass it everywhere. I keep that in its own module, judge.py, and read the key from the environment so no credential lands in git.

This file shows how to call OpenAI directly, or Gemini through Google’s OpenAI-compatible endpoint.

# judge.py
"""Build the judge LLM and embedding model that every metric shares.

Uses OpenAI by default (reads OPENAI_API_KEY). Set GEMINI_API_KEY instead to
judge with Gemini through Google's OpenAI-compatible endpoint.
"""

import os

from openai import AsyncOpenAI
from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory

GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai/"


def judge_model_name() -> str:
    """The judge model to use (override it with the JUDGE_MODEL variable)."""
    default = "gemini-3.5-flash-lite" if os.environ.get("GEMINI_API_KEY") else "gpt-4o-mini"
    return os.environ.get("JUDGE_MODEL", default)


def build_judge():
    """Return (llm, embeddings), both using one OpenAI-compatible client."""
    if os.environ.get("GEMINI_API_KEY"):
        client = AsyncOpenAI(
            api_key=os.environ["GEMINI_API_KEY"],
            base_url=GEMINI_BASE_URL,
            max_retries=5,
        )
        embedding_model = "gemini-embedding-001"
        # Gemini 3.x models spend output tokens thinking before they answer
        settings = {"max_tokens": 4096, "reasoning_effort": "low"}
    else:
        client = AsyncOpenAI(max_retries=5)
        embedding_model = "text-embedding-3-small"
        settings = {"max_tokens": 4096}

    llm = llm_factory(judge_model_name(), client=client, **settings)
    embeddings = embedding_factory("openai", model=embedding_model, client=client)
    return llm, embeddings

Two details in that file can trip up your first run:

• llm_factory() in 0.4 needs a client object passed in, and passing only a model name throws an error.
• Any extra settings you pass to llm_factory() go straight to the model call, which is how I raise max_tokens above the 1,024 default.

Which judge model should I use?

Use the cheapest model that produces stable scores on a sample you have hand-checked, and then pin it. For a large evaluation suite the volume adds up fast: faithfulness makes 2 judge calls per sample, answer relevancy makes 3 plus 2 embedding calls, context precision makes 1 per retrieved chunk, and context recall makes 1.

Six samples across 4 metrics came to 62 requests in a single run: 50 judge calls plus 12 embedding calls. Here is what 3 current small judge models cost at each vendor’s list price on October 19, 2026:

The default in judge.py is still gpt-4o-mini, an older OpenAI model that is not listed in the main price table on OpenAI’s pricing page. Set the JUDGE_MODEL variable to switch.

One catch if you want Claude as the judge: answer relevancy also needs an embedding model, and Anthropic does not offer one. Its docs point to Voyage AI, so you would pair Claude with embeddings from another provider.

Pinning matters more than the choice. Swapping the judge model shifts scores even when your pipeline did not change, which destroys the trend line you are about to build in the tracking section.

Put the model name in judge.py under version control and treat a judge upgrade like any other breaking change: reset every threshold against the new judge’s scores.

How Do I Build a RAG Evaluation Dataset?

Each test case in the dataset is called a sample. A sample is more than an answer: it records one full trip through the pipeline, in 4 fields:

Ragas stores each sample as a SingleTurnSample and the whole set as an EvaluationDataset. The reference is optional, but the retrieval metrics compare against it, so I always include it in the curated set even though the reference-free metrics ignore it.

I built 6 samples for a fictional coffee roaster, Northwind Coffee. Each metric section below adds the sample that metric should catch.

Start build_dataset.py with the first 3 samples: a clean answer, a deliberate hallucination, and a second clean answer that makes several claims at once.

# build_dataset.py
"""Hand-build a small evaluation set and save it as JSON Lines."""

import json

from ragas import EvaluationDataset, SingleTurnSample

# Chunks of Northwind's help-center pages, exactly as a retriever would
# return them to the generator.
REFUNDS = (
    "If you are not happy with your coffee, contact support within 14 days of "
    "the delivery date for a full refund. You do not need to return the bag."
)
REFUND_TIMING = (
    "Refunds are issued to the original payment method and appear within 5 to "
    "7 business days. Gift orders are refunded as store credit."
)
SHIPPING_US = (
    "Standard shipping within the contiguous United States takes 3 to 5 "
    "business days and costs $6.00. Standard shipping is free on orders of "
    "$40.00 or more."
)
PAUSE = (
    "You can pause a subscription for up to 3 months from your account page. "
    "Paused subscriptions keep their pricing."
)
SUBSCRIPTION_PERKS = (
    "Subscribers save 10% on every bag and always get free standard shipping, "
    "regardless of order size."
)

samples = [
    # 1. Clean: grounded answer, relevant chunks ranked first.
    SingleTurnSample(
        user_input="How long do I have to ask for a refund on a bag of coffee?",
        retrieved_contexts=[REFUNDS, REFUND_TIMING, SHIPPING_US],
        response=(
            "You have 14 days from the delivery date to contact support for a "
            "full refund, and you do not need to send the bag back."
        ),
        reference=(
            "Contact support within 14 days of the delivery date for a full "
            "refund; the bag does not need to be returned."
        ),
    ),
    # 2. Deliberate hallucination: the pause limit is 3 months, not 6, and
    #    the free bag does not exist anywhere in the docs.
    SingleTurnSample(
        user_input="Can I pause my subscription?",
        retrieved_contexts=[PAUSE, SUBSCRIPTION_PERKS],
        response=(
            "Yes. You can pause your subscription for up to 6 months from your "
            "account page, and you get a free bag when you resume."
        ),
        reference=(
            "Yes, a subscription can be paused for up to 3 months from the "
            "account page, and paused subscriptions keep their pricing."
        ),
    ),
    # 3. Clean: multi-fact answer fully supported by the retrieved chunks.
    SingleTurnSample(
        user_input="What do subscribers get that regular customers do not?",
        retrieved_contexts=[SUBSCRIPTION_PERKS, PAUSE],
        response=(
            "Subscribers save 10% on every bag and get free standard shipping "
            "on every order, with no minimum order size."
        ),
        reference=(
            "Subscribers save 10% on every bag and always get free standard "
            "shipping regardless of order size."
        ),
    ),
]

dataset = EvaluationDataset(samples=samples)
dataset.to_jsonl("eval_dataset.jsonl")
print(dataset)

# Save the questions and reference answers separately. The CI section rebuilds
# retrieved_contexts and response from your real pipeline, so it only needs these.
with open("eval_questions.jsonl", "w", encoding="utf-8") as file:
    for sample in samples:
        row = {"user_input": sample.user_input, "reference": sample.reference}
        file.write(json.dumps(row) + "\n")

Each retrieved context is a chunk of Northwind’s help-center pages, written into the script exactly as a retriever would hand it to the generator. Run uv run python build_dataset.py to write eval_dataset.jsonl and eval_questions.jsonl, and rerun it each time a later section adds a sample.

Saving to JSON Lines matters because every later script (the metric runs, the CI gate) loads the same eval_dataset.jsonl with EvaluationDataset.from_jsonl().

In a real project, the retrieved_contexts and response fields come from running your actual pipeline over the questions, and only user_input and reference are hand-written. The CI section shows how to automate that step.

What makes a good evaluation set

The 6 samples this article ends up with are enough to learn the metrics, but not to trust a threshold. Aim for around 100 samples before you rely on one; the tracking section shows why. How many you need depends on how strict your use case is, so I focus on a good mix rather than a specific number:

• Real query distribution. Pull questions from support logs or search analytics, not from your own head. Users phrase questions in ways you do not expect, and rarely match your FAQ wording.
• Edge cases the docs handle badly. Questions whose answer spans 2 documents, questions with a date or a number in them, and questions that use a synonym the docs do not.
• Unanswerable questions. Sample 6, which you will add in the context precision and recall section, is one example. Your docs cannot answer everything, and the correct behavior is to say so. If you never test that, you never notice the day the model starts guessing.
• Known past failures. Every hallucination a user reported becomes a permanent sample. That is how an evaluation set grows into a regression test suite.

What Do the 4 Core Ragas Metrics Measure?

The 4 core Ragas metrics are faithfulness and answer relevancy for the generator, and context precision and context recall for the retriever. All 4 return a score between 0 and 1. The useful part is what each number tells you to fix.

Faithfulness: catching hallucinations

Faithfulness measures how much of an answer is backed up by the retrieved chunks. The judge splits the answer into individual claims, then checks each claim against the chunks: a score of 1.0 means every claim is supported, and 0.0 means none are.

Faithfulness does not need a reference answer, so you can run it on live production traffic, not just your test set. Rerun it whenever you change the prompt, the model, or how much context the model sees, because those are the changes that most often introduce hallucinations.

The quickest way to see it in action is on a single answer, no dataset required:

# quick_check.py
"""Score a single response for faithfulness"""

import asyncio

from ragas.metrics.collections import Faithfulness

from judge import build_judge

judge_llm, _ = build_judge()
faithfulness = Faithfulness(llm=judge_llm)

result = asyncio.run(
    faithfulness.ascore(
        user_input="Can I pause my subscription?",
        response=(
            "Yes. You can pause for up to 6 months, and you get a free bag "
            "when you resume."
        ),
        retrieved_contexts=[
            "You can pause a subscription for up to 3 months from your account "
            "page. Paused subscriptions keep their pricing."
        ],
    )
)
print(f"faithfulness = {result.value:.2f}")

Run it with uv run python quick_check.py and it prints faithfulness = 0.00, because neither claim in the response (6 months, a free bag) is supported by the context. In 0.4 you score with each metric’s ascore() method, which returns a result whose .value is the score.

Scored across the dataset so far, the planted hallucination in sample 2 gets 0.0, as it should, and both clean samples get 1.0. The full results for every metric are in the multi-metric section below.

Answer relevancy: catching drift

Answer relevancy measures whether the answer actually addresses the question. The judge reads the answer and writes 3 questions it would be a good answer to, then Ragas checks how close in meaning those questions are to the original one, using an embedding model. If the answer dodges the question, the judge marks it as noncommittal and it scores 0.

Ragas calls the class AnswerRelevancy (older examples call it ResponseRelevancy, and the docs page is titled Response Relevancy), and it needs both the judge LLM and the embedding model.

A low relevancy score with a high faithfulness score usually points at the generator or the prompt: the model had the right context and stayed grounded in it, but answered a different question than the one asked. Check context precision and recall first, though, because a retriever that returns off-topic chunks can produce the same pattern.

This sample shows the generator case. The retriever returns the right shipping chunk, the answer stays grounded, and it talks about roast dates instead of shipping cost. Add the new chunk to the chunks at the top of build_dataset.py:

ROAST_DATE = (
    "Every bag is roasted to order and shipped within 48 hours of roasting. "
    "The roast date is printed on the bottom of the bag."
)

Then add the new sample at the end of the samples list:

    # 4. Drift: retrieval was fine and the answer is grounded, but it never
    #    addresses the question that was asked.
    SingleTurnSample(
        user_input="How much does standard shipping cost in the US?",
        retrieved_contexts=[SHIPPING_US, ROAST_DATE],
        response=(
            "Every bag is roasted to order and shipped within 48 hours of "
            "roasting, and the roast date is printed on the bottom of the bag."
        ),
        reference=(
            "Standard shipping in the contiguous United States costs $6.00 and "
            "is free on orders of $40.00 or more."
        ),
    ),

Each sample gets one relevancy score: how closely the questions the judge wrote from the answer match the question the sample actually asked, where 1.0 is a perfect match.

Context precision and recall: diagnosing retrieval

Context precision checks whether the useful chunks are ranked near the top of what the retriever returned, and context recall checks whether the retriever found everything needed to write the correct answer.

Precision gives a useful chunk more credit the higher it is ranked, so a useful chunk in first place counts for more than the same chunk in third. Recall is the share of facts in the reference answer that at least one retrieved chunk supports.

Both need the reference field. In the collections API the classes are ContextPrecisionWithReference and ContextRecall, and both take only the judge LLM. The Ragas docs also use the shorter name ContextPrecision for the reference-based version, so you may see either.

So far every sample retrieves the right chunks first, so neither metric has anything to catch. Let’s add 2 samples that break retrieval in different ways: one where the right chunk is retrieved but ranked last, and one where nothing relevant is retrieved at all, so the model correctly declines. As before, add the chunks to the top of the file:

SHIPPING_CANADA = (
    "We ship to Canada. Canadian orders take 7 to 10 business days and cost "
    "$18.00. We do not currently ship outside the United States and Canada."
)
BREWING = (
    "For pour-over, we recommend a ratio of 1 gram of coffee to 16 grams of "
    "water, a medium-fine grind, and water at 200 degrees Fahrenheit."
)

Then add the samples to the end of the list:

    # 5. Ranking problem: the right chunk is retrieved, but it is ranked last.
    SingleTurnSample(
        user_input="Does Northwind ship to Canada, and how long does it take?",
        retrieved_contexts=[BREWING, ROAST_DATE, SHIPPING_CANADA],
        response=(
            "Yes, Northwind ships to Canada. Canadian orders take 7 to 10 "
            "business days and cost $18.00."
        ),
        reference="Yes. Canadian orders take 7 to 10 business days and cost $18.00.",
    ),
    # 6. Retrieval miss: nothing about the warranty was retrieved, so the
    #    model correctly declines. Recall will be 0 here.
    SingleTurnSample(
        user_input="How long is the warranty on the Baratza Encore grinder?",
        retrieved_contexts=[BREWING, SHIPPING_US],
        response=(
            "I do not have information about the grinder warranty in the "
            "documents available to me."
        ),
        reference=(
            "All equipment, including the Baratza Encore grinder, ships with a "
            "2-year manufacturer warranty covering defects in materials and "
            "workmanship."
        ),
    ),

Ranking matters because most pipelines pass only the top few chunks to the model, so a useful chunk in 6th place of a top-5 retriever might as well not exist.

Sample 5 puts its one useful chunk last of 3, and precision comes back at 0.333 since the only useful information is ranked last. Sample 6, the retrieval miss, scored 0.0 on both precision and recall.

Sample 6 also teaches something about faithfulness. The model correctly refused, yet that refusal scored 1.0 on faithfulness in some runs and 0.0 in others, with nothing changing in between. A statement like “I do not have information about the warranty” is neither supported nor contradicted by the context, and the judge flips a coin on it.

If your pipeline is supposed to refuse, score refusals separately rather than letting them drag faithfulness around. Ragas 0.4 removed AspectCritic from the collections API, so the replacement is a custom check built with the @discrete_metric decorator (see the v0.3 to v0.4 migration guide).

Read the 2 retrieval numbers together and they tell you where to spend your next week:

The “low precision, high recall” row is the most common one I see in practice, and it is the cheapest to fix.

Our RankGPT as a Re-Ranking Agent for RAG tutorial adds an LLM reranker on top of an existing retriever, and our Boost LLM Accuracy with RAG and Reranking tutorial shows another reranking setup. Our 5 key techniques for improving RAG performance guide covers chunking and query rewriting for the other rows.

Rerun the 2 retrieval metrics after each change and the table tells you whether it worked.

How Do I Run a Full Multi-Metric Evaluation?

A full multi-metric evaluation runs all 4 metrics over every sample in parallel and collects the scores, with per-row values for drilling into failures and column means for your dashboard.

In Ragas 0.2 and 0.3 you did this with evaluate(dataset, metrics=[...], run_config=RunConfig(max_workers=8)). In 0.4.3 that function still exists, but it warns that it is deprecated and refuses the new metrics outright, because they no longer inherit from the old Metric class it expects.

The replacement is the @experiment decorator, which runs your pipeline and your metrics together over a Dataset and writes results to CSV. That is the right tool once you have a pipeline to call.

For scoring a static dataset, I find a small helper clearer: it builds the 4 metrics once for every script that needs them, and it keeps the parallel calls and error handling in code I can read:

# score.py
"""Score an EvaluationDataset with the 4 core Ragas metrics."""

import asyncio
import inspect
import math

import pandas as pd
from ragas.metrics.collections import (
    AnswerRelevancy,
    ContextPrecisionWithReference,
    ContextRecall,
    Faithfulness,
)


def build_metrics(judge_llm, judge_embeddings) -> list:
    """The 4 core metrics, all using the same judge."""
    return [
        Faithfulness(llm=judge_llm),
        AnswerRelevancy(llm=judge_llm, embeddings=judge_embeddings),
        ContextPrecisionWithReference(llm=judge_llm),
        ContextRecall(llm=judge_llm),
    ]


async def score_sample(sample, metrics, limit):
    """Run every metric on one sample and return that sample's row of scores."""
    row = {"user_input": sample.user_input}
    for metric in metrics:
        # Each metric needs different fields (recall wants the reference,
        # faithfulness wants the response), so pass only the ones it asks for.
        wanted = inspect.signature(metric.ascore).parameters
        inputs = {field: getattr(sample, field) for field in wanted}
        async with limit:
            try:
                result = await metric.ascore(**inputs)
                row[metric.name] = result.value
            except Exception as error:
                # One failed judge call should not sink the whole run
                print(f"{metric.name} failed on {sample.user_input!r}: {error}")
                row[metric.name] = math.nan
    return row


def score_dataset(dataset, metrics, max_concurrency=8):
    """Score every sample: one row per sample, one column per metric."""

    async def score_all_samples():
        limit = asyncio.Semaphore(max_concurrency)  # cap parallel judge calls
        tasks = [score_sample(sample, metrics, limit) for sample in dataset.samples]
        return await asyncio.gather(*tasks)

    return pd.DataFrame(asyncio.run(score_all_samples()))

The one non-obvious line is the inspect.signature() call. The 4 metrics want different inputs (faithfulness needs the response, recall needs the reference instead), so the helper reads which arguments each metric’s ascore() accepts and passes exactly those fields from the sample, which lets a single loop serve every metric.

The semaphore caps how many judge calls run at once, and a metric that still fails becomes an empty (NaN) cell instead of aborting the whole run. If you run this inside a Jupyter notebook, asyncio.run fails because a loop is already running. Call nest_asyncio.apply() first (Ragas already depends on it) or await the coroutine directly.

With the helper in place, the full run is short:

# run_all.py
"""Run all four core metrics and save per-sample scores to scores.csv."""

import pandas as pd
from ragas import EvaluationDataset

from judge import build_judge
from score import build_metrics, score_dataset

judge_llm, judge_embeddings = build_judge()
dataset = EvaluationDataset.from_jsonl("eval_dataset.jsonl")
metrics = build_metrics(judge_llm, judge_embeddings)

scores = score_dataset(dataset, metrics, max_concurrency=4)
scores.to_csv("scores.csv", index=False)

pd.set_option("display.max_colwidth", 45)
pd.set_option("display.width", 160)
print(scores.sort_values("faithfulness").round(3))
print("\nMeans:")
print(scores.drop(columns="user_input").mean().round(3))

Here are the scores from my first run, sorted by faithfulness and rounded to 3 decimals, with the means in the last row:

Later runs of the same dataset scored faithfulness lower because of the refusal-sample flip I cover in the tracking section.

Sorting by faithfulness helps you pick out the likeliest hallucinations, and the planted one is at the top. The means in the last row are your dashboard numbers.

Read across the rows and each planted failure shows up in a different place. The hallucination drops faithfulness only. The retrieval miss zeroes relevancy, precision, and recall together. The ranking problem cuts precision to 0.333, and the drift shows up as the lowest relevancy among the answered questions.

One oddity: the Canada answer is fully supported by its chunk, yet it scored 0.667 on faithfulness. That is judge noise, which I come back to in the tracking section.

Answer relevancy compares meaning with embeddings, so even a good answer rarely scores near 1.0: the refund answer scored 0.746, while the off-topic shipping answer scored 0.572. Judge relevancy by the gap between samples and by the trend over time, not by the absolute number.

How Do I Add a RAG Quality Gate to CI?

A RAG quality gate is a test script that runs the evaluation, compares each mean score against a threshold, and exits with an error code when any metric falls short, which tells the CI system to block the merge.

One catch first. The hand-built dataset from earlier has its responses and retrieved chunks written in by hand, so scoring it can never reflect a change to your prompt or retriever. In CI you need to rerun the real pipeline over your saved questions and then score what it produced. Two small scripts do that.

# pipeline.py
"""The bridge between the evaluation scripts and your own RAG pipeline."""


def answer(question: str) -> tuple[list[str], str]:
    """Return (retrieved_chunks, generated_answer) for one question.

    Replace the body with a call to your own retriever and generator. The
    chunks must be in the order your generator receives them, because context
    precision depends on ranking.
    """
    raise NotImplementedError("Connect this function to your RAG pipeline.")

# refresh_dataset.py
"""Run the real pipeline over the saved questions and rebuild eval_dataset.jsonl."""

import json

from ragas import EvaluationDataset, SingleTurnSample

from pipeline import answer


def main():
    samples = []
    with open("eval_questions.jsonl", encoding="utf-8") as file:
        for line in file:
            row = json.loads(line)
            retrieved_contexts, response = answer(row["user_input"])
            samples.append(
                SingleTurnSample(
                    user_input=row["user_input"],
                    retrieved_contexts=retrieved_contexts,
                    response=response,
                    reference=row["reference"],
                )
            )
    EvaluationDataset(samples=samples).to_jsonl("eval_dataset.jsonl")
    print(f"Rebuilt eval_dataset.jsonl with {len(samples)} samples")


if __name__ == "__main__":
    main()

pipeline.py is the only file you write yourself: replace the body of answer() with a call to your retriever and generator. refresh_dataset.py reads the questions and reference answers that build_dataset.py saved, runs the pipeline over each question, and rewrites eval_dataset.jsonl.

The gate itself is a dictionary of thresholds and a loop. The thresholds below were calibrated on the hand-built example dataset, so recalibrate them on your own pipeline’s output before you rely on them.

# quality_gate.py
"""Fail the build when any aggregate Ragas score drops below its threshold."""

import os
import random
import sys

import numpy as np
import pandas as pd
from ragas import EvaluationDataset

from judge import build_judge
from score import build_metrics, score_dataset

# Set each threshold just below the lowest mean you have seen across repeated
# runs, not below one lucky run. With a small dataset a single flaky verdict
# moves a mean by 1 / n_samples, so a thin margin makes the build flaky.
THRESHOLDS = {
    "faithfulness": 0.60,
    "answer_relevancy": 0.60,
    "context_precision_with_reference": 0.65,
    "context_recall": 0.75,
}

RUNS = int(os.environ.get("GATE_RUNS", "3"))      # judge runs to average
SAMPLE = int(os.environ.get("GATE_SAMPLE", "0"))  # rows to score, 0 means all


def load_dataset():
    dataset = EvaluationDataset.from_jsonl("eval_dataset.jsonl")
    if 0 < SAMPLE < len(dataset.samples):
        # A fixed seed means every pull request scores the same subset
        rows = random.Random(0).sample(dataset.samples, SAMPLE)
        dataset = EvaluationDataset(samples=rows)
    return dataset


def average_runs(frames):
    """Average each sample's scores across runs. A NaN in any run stays NaN."""
    numbers = [frame.drop(columns="user_input") for frame in frames]
    averaged = np.mean([frame.to_numpy() for frame in numbers], axis=0)
    result = pd.DataFrame(averaged, columns=numbers[0].columns)
    result.insert(0, "user_input", frames[0]["user_input"])
    return result


def main():
    judge_llm, judge_embeddings = build_judge()
    dataset = load_dataset()
    metrics = build_metrics(judge_llm, judge_embeddings)

    frames = [score_dataset(dataset, metrics, max_concurrency=4) for _ in range(RUNS)]
    scores = average_runs(frames)
    scores.to_csv("scores.csv", index=False)  # track_scores.py reads this file

    # skipna=False: a failed judge call must fail the gate, not shrink the sample
    means = scores.drop(columns="user_input").mean(skipna=False)

    failed = []
    for metric_name, threshold in THRESHOLDS.items():
        actual = means[metric_name]
        # Test the pass condition once and fail on everything else. NaN is
        # neither >= nor < a threshold, so a judge outage must fail the gate
        # rather than slip between the two comparisons.
        ok = actual >= threshold
        print(f"{'PASS' if ok else 'FAIL'}  {metric_name:<34} {actual:.3f} (min {threshold:.2f})")
        if not ok:
            failed.append(metric_name)

    if failed:
        print(f"\nQuality gate failed on: {', '.join(failed)}")
        sys.exit(1)
    print("\nQuality gate passed.")


if __name__ == "__main__":
    main()

Against my baseline, all 4 means clear their thresholds, and the script exits with code 0, which lets the merge through:

I set the faithfulness threshold at 0.60 rather than 0.70 because of the judge noise I cover in the tracking section. With 6 samples, one flaky verdict moves a mean by 0.17, and a later run of the unchanged pipeline scored 0.61. The gate also runs the judge 3 times and averages the scores (GATE_RUNS), which smooths out single-verdict flips at 3 times the judge cost.

One design choice is to fail on not (actual >= threshold) rather than actual < threshold. A comparison against NaN (“not a number”) is always False, so a check written as actual < threshold would pass the build whenever the judge fails.

The skipna=False in the mean covers the partial case. pandas skips NaN by default, so a few failed judge calls would quietly shrink the sample instead of failing the gate. A transient judge error now fails the build, so rerun the job before you debug.

Thresholds are tricky to tune. Set them slightly below the lowest mean you have seen across repeated runs, not at some aspirational 0.9, or the gate blocks every pull request and the team stops trusting CI. Reset them whenever you change the judge model or add more than a handful of samples.

Wiring it into GitHub Actions is a short workflow. The workflow syntax reference covers the on.pull_request path filters and the schedule cron used here:

# .github/workflows/rag-eval.yml
name: RAG quality gate

on:
  pull_request:
    paths:
      - "prompts/**"
      - "retriever/**"
      - "judge.py"
      - "eval_questions.jsonl"
  schedule:
    - cron: "0 3 * * *"  # full suite nightly at 03:00 UTC

jobs:
  ragas:
    runs-on: ubuntu-latest
    env:
      # Use the same judge you calibrated the thresholds with
      GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
      JUDGE_MODEL: gemini-3.5-flash-lite
      # Score a fixed 30-row subset on pull requests, everything on the nightly run
      GATE_SAMPLE: ${{ github.event_name == 'pull_request' && '30' || '0' }}
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
      - run: uv sync
      - name: Rebuild the dataset with the current pipeline
        run: uv run python refresh_dataset.py
      - name: Run Ragas quality gate
        run: uv run python quality_gate.py
      - name: Log scores
        if: ${{ always() && hashFiles('scores.csv') != '' }}
        run: uv run python track_scores.py
      - name: Keep the scores
        if: ${{ always() && hashFiles('scores.csv') != '' }}
        uses: actions/upload-artifact@v7
        with:
          name: ragas-scores
          path: |
            scores.csv
            ragas_history.jsonl

A few details in that workflow matter:

• The judge is pinned with JUDGE_MODEL and its matching API key. Use the same judge you calibrated the thresholds with, or CI scores will not be comparable with your tracked history.
• On pull requests, GATE_SAMPLE scores a fixed random subset of 30 rows, so a review is not waiting 10 minutes on judge calls. The nightly cron scores everything.
• The path filter runs the gate only when the prompt, retriever, judge setup, or questions change.
• Each run uploads scores.csv and ragas_history.jsonl as an artifact, because the runner’s disk disappears when the job ends.

Run the full suite on the nightly cron, where nobody is waiting. A failing nightly run shows up as a failed workflow, and GitHub’s notification settings or a Slack step can tell the owner in the morning. GitLab CI has the same shape with rules: changes: and a pipeline schedule.

How Do I Generate Synthetic Test Data With Ragas?

Ragas generates synthetic test questions by reading your documents, pulling out their main topics and facts, linking chunks that are related, and then writing questions of different kinds from those links.

The test data generation docs split queries into single-hop (one chunk can answer) and multi-hop (the answer needs several chunks), and each of those can be specific or abstract. The default mix uses 3 of those 4 combinations in equal shares:

• Single-hop specific questions that one chunk can answer.
• Multi-hop abstract questions that need reasoning across several chunks.
• Multi-hop specific questions that combine a fact from each of several chunks.

Ragas only uses the synthesizers your documents can support, so a small document set can produce a different mix.

The generator needs real documents to read. Point it at your own documentation as markdown files. For this article, I wrote 5 short help-center pages for Northwind (about 600 words in total), and each one looks like this:

<!-- docs/subscription.md -->
# Subscriptions

A Northwind subscription delivers a fresh bag on a schedule you choose: every 1, 2, or 4 weeks. Subscribers save 10% on every bag and always get free standard shipping, regardless of order size.

You can pause a subscription for up to 3 months from your account page. Paused subscriptions keep their pricing. You can also skip a single delivery up to 48 hours before it is scheduled to ship.

Cancel at any time from the account page. Cancellations take effect immediately and no further bags are charged. There is no cancellation fee.

Subscribers can change their roast, grind, or bag size at any time. Changes made less than 48 hours before a scheduled shipment apply to the following delivery.

Your own docs will produce different questions from the ones below, which is the point: the generator writes questions about whatever your knowledge base actually says.

The TestsetGenerator takes the same judge LLM and embedding model we already built. I load the docs with LangChain’s DirectoryLoader, using TextLoader explicitly so the markdown files do not pull in the heavy unstructured dependency.

Mapping out your documents fires many judge calls at once, so I also limit how many run in parallel with RunConfig:

# generate_testset.py
"""Generate synthetic question-answer pairs from the documents in ./docs."""

import pandas as pd
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from ragas import RunConfig
from ragas.testset import TestsetGenerator

from judge import build_judge

judge_llm, judge_embeddings = build_judge()

loader = DirectoryLoader(
    "docs", glob="**/*.md", loader_cls=TextLoader, loader_kwargs={"encoding": "utf-8"}
)
docs = loader.load()
print(f"Loaded {len(docs)} documents")

generator = TestsetGenerator(llm=judge_llm, embedding_model=judge_embeddings)
testset = generator.generate_with_langchain_docs(
    docs,
    testset_size=8,
    run_config=RunConfig(max_workers=4),  # keep the knowledge-graph build polite
)

frame = testset.to_pandas()
frame.to_csv("synthetic_testset.csv", index=False)

pd.set_option("display.max_colwidth", 70)
print(frame[["user_input", "synthesizer_name"]])

The script loads all 5 documents and writes these 8 questions. The questions track the documents, and some contain deliberate typos:

The table shows simplified labels; the synthesizer_name column in the output holds the full synthesizer name. Eight questions from 5 documents took about 4.5 minutes.

The typos and the keyword-style query in question 2 are deliberate: the generator varies the style and length of each query so the set looks like search-box traffic rather than exam questions.

You will probably want to look over the questions to make sure they are usable and relevant. I would use the output as a first draft. In every synthetic set I have reviewed, some questions are malformed (they reference “the document” or ask about a heading), some reference answers drift from what the source actually says, and the phrasing is more formal than any real user’s.

Rerun the generator when you add a section to the docs and you get candidate questions for the new material, at the cost of another round of judge calls.

Each row has a user_input, a generated reference answer, and reference_contexts, the chunks the question was built from. Check the reference against those chunks, because that is where drift shows up.

To turn a reviewed row into a SingleTurnSample, run your pipeline over the user_input to get the retrieved_contexts and response. Append the reviewed user_input and reference pairs to eval_questions.jsonl and refresh_dataset.py does that for you.

If your knowledge base is a graph rather than a pile of documents, the same principle applies with a different retriever. Our Using a Knowledge Graph to Implement a RAG Application tutorial and our Graph RAG with LangChain and Neo4j course both produce pipelines you can evaluate with exactly the 4 metrics above, because the metrics only see the retrieved text, not how it was retrieved.

How Do I Track Ragas Scores Over Time?

Tracking Ragas scores over time means appending the average scores, the git commit hash, the judge model, and a timestamp to a log file after every evaluation run, so a regression shows up as a dip on a line chart instead of a surprise in production.

A JSON Lines file is probably the best bet until you need a shared experiment tracker for several people. track_scores.py reads scores.csv, which quality_gate.py and run_all.py both write, so it never triggers a second round of judge calls.

# track_scores.py
"""Append the latest aggregate scores, commit hash, and timestamp to a log."""

import json
import subprocess
from datetime import datetime, timezone

import pandas as pd

from judge import judge_model_name

HISTORY_FILE = "ragas_history.jsonl"


def current_commit():
    """Return the short git hash, or 'no-git' outside a repository."""
    try:
        return subprocess.check_output(
            ["git", "rev-parse", "--short", "HEAD"],
            text=True, stderr=subprocess.DEVNULL,
        ).strip()
    except (subprocess.CalledProcessError, FileNotFoundError):
        return "no-git"


def main():
    scores = pd.read_csv("scores.csv")
    means = scores.drop(columns="user_input").mean().round(4)

    record = {
        "timestamp": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "commit": current_commit(),
        "judge_model": judge_model_name(),
        "n_samples": len(scores),
        **means.to_dict(),
    }
    with open(HISTORY_FILE, "a", encoding="utf-8") as history:
        history.write(json.dumps(record) + "\n")
    print(json.dumps(record, indent=2))


if __name__ == "__main__":
    main()

Running uv run python track_scores.py appends one line to the log and prints it:

{
  "timestamp": "2026-09-18T03:23:51+00:00",
  "commit": "85834f4",
  "judge_model": "gemini-3.5-flash-lite",
  "n_samples": 6,
  "faithfulness": 0.7778,
  "answer_relevancy": 0.6678,
  "context_precision_with_reference": 0.7222,
  "context_recall": 0.8333
}

In CI each job starts with an empty disk, so ragas_history.jsonl only ever holds that run’s record. To keep a trend, download the uploaded artifacts and append them to one file, or commit the file to a metrics branch or object storage.

Plotting the history is pretty easy to do. Try the following script:

# plot_history.py
"""Plot every logged Ragas run so regressions show up as a dip in a line."""

import matplotlib.pyplot as plt
import pandas as pd

history = pd.read_json("ragas_history.jsonl", lines=True)
history["timestamp"] = pd.to_datetime(history["timestamp"])
history["run"] = history["commit"] + "\n" + history["timestamp"].dt.strftime("%m-%d %H:%M")

METRICS = {
    "faithfulness": "#2a78d6",
    "answer_relevancy": "#eb6834",
    "context_precision_with_reference": "#1baf7a",
    "context_recall": "#eda100",
}

fig, ax = plt.subplots(figsize=(9, 4.5))
for metric, color in METRICS.items():
    ax.plot(history["run"], history[metric], color=color, linewidth=2,
            marker="o", markersize=6, label=metric)
    ax.annotate(f"{history[metric].iloc[-1]:.2f}",
                (history["run"].iloc[-1], history[metric].iloc[-1]),
                xytext=(6, 0), textcoords="offset points", va="center",
                fontsize=9, color="#333333")

ax.set_ylim(0, 1.05)
ax.set_ylabel("mean score")
ax.set_title("Ragas scores by commit")
ax.grid(axis="y", color="#e5e5e5")
ax.spines[["top", "right"]].set_visible(False)
ax.legend(loc="lower left", frameon=False)
fig.tight_layout()
fig.savefig("ragas_history.png", dpi=150)
print("Saved ragas_history.png")

You will get a chart that looks something like this:

Mean Ragas scores logged per commit by track_scores.py. Image by author.

Nothing in the pipeline changed between those commits (the second one only changed the judge client’s connection settings), yet faithfulness fell from 0.78 to 0.61 because the refusal in sample 6 flipped from 1.0 to 0.0.

So there is real judge noise to watch for until you have a large number of samples. With 6 samples, 1 flaky verdict moves the mean by 0.17. With 100 samples the same flip moves it by 0.01, so a big change is much more likely to be a real regression.

Now, with every pull request or commit, you can run the gate, log the scores, and look at the chart.

If you are weighing a bigger architectural change, our RAG vs. fine-tuning tutorial frames the trade-off, and the same 4 metrics work as the before-and-after measurement for either path.

What Are the Limits of LLM-as-a-Judge Scores?

LLM-as-a-judge scores are estimates: they have known biases and can change from run to run, but they scale to evaluation volumes that human review cannot.

Ragas never asks the judge to compare 2 answers, so position bias (favoring whichever answer comes first) mostly does not apply. Verbosity bias (favoring longer answers) and self-preference bias (judges favoring answers from their own model) can still matter. Zheng et al. (2023) describe all 3 biases, and self-preference is why I would not use the same model as generator and judge.

The run-to-run variation is the part I want you to see with your own eyes, because it changes how you set thresholds. Across runs of the same 6 samples with the same pinned judge, the refusal answer in sample 6 scored 0.0 in one run and 1.0 in another on faithfulness, and the fully grounded sample 5 scored 0.667 in the run shown above.

Per-sample scores from a cheap judge are noisy. Means over 100 samples are far less noisy, and the CI gate should always test the mean.

Three practices contain the noise:

1. Pin the judge model and log it, as judge.py and track_scores.py do above, so a judge change is never mistaken for a pipeline change.
2. Spot-check the judge against humans on 20 to 30 samples when you set it up. If the judge and a colleague disagree on more than a handful, switch to a stronger model before trusting a single threshold.
3. Score refusals separately from faithfulness, since an “I do not know” answer has no claims for the judge to verify and the score is arbitrary.

For a broader tour of evaluation approaches beyond RAG, including the human evaluation methods that the judge is standing in for, our LLM evaluation guide covers the metrics and methodologies. Our LLM as a judge tutorial goes deeper on building a judge for RAG pipelines.

Final Thoughts

The point of this pipeline is that it lets us say things like “the retriever is fine, the prompt regressed” or “the reranker fixed precision but recall is still 0.6”, with numbers behind the claim.

Faithfulness and answer relevancy watch the generator: faithfulness catches invented facts, and answer relevancy catches answers that drift off the question. Context precision and recall watch the retriever and tell you whether it pulls the right information. Because 2 of the 4 need no reference answer, you can run them on production traffic where no reference exists.

Standard text metrics measure word overlap, which says nothing about an invented refund policy, and a single end-to-end number hides which half of the pipeline broke. A framework that checks the 2 failure points separately can see it, and turning it from a one-off audit into a CI gate with a history file is what gets your team to act on the numbers.

If you have not built the pipeline yet, start with our Retrieval-Augmented Generation with LangChain course and then point this article’s pipeline.py at what you build.

For pipelines that go beyond plain text retrieval, our End-to-End RAG with Weaviate course, our multimodal RAG guide, and our agentic RAG tutorial build systems you can score the same way, with one caveat: the 4 core metrics only read text. A pipeline that retrieves images needs Ragas’s separate multimodal metrics, and an agentic pipeline needs each retrieval step scored as its own question, chunks, and answer.