• October 25, 2026
• 9 min read

Ragas — RAG Metrics¶

A RAG system has two parts that fail differently: retrieval (did the right passages come back, in a useful order?) and generation (did the answer stay inside those passages and address the question?). Ragas scores each part separately, so a regression points at the retriever, the prompt or the model instead of at "the pipeline".

Import metrics from ragas.metrics.collections. Retrieval metrics with no LLM (NonLLMContext*, IDBasedContext*) exist only as legacy classes in ragas.metrics and are covered at the end.

Metric Map¶

Legacy names: ResponseRelevancy is AnswerRelevancy, LLMContextRecall is ContextRecall, LLMContextPrecisionWithReference / LLMContextPrecisionWithoutReference are the two context precision classes. Most collections metrics raise ValueError (for example response is missing) when a required field is empty.

import asyncio

from openai import AsyncOpenAI
from ragas import EvaluationDataset
from ragas.embeddings.base import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness,
)

from evals.harness import ascore_dataset          # the helper from page 01


async def main(dataset: EvaluationDataset):
    client = AsyncOpenAI()
    llm = llm_factory("gpt-4o-mini", client=client, temperature=0)
    emb = embedding_factory("openai", model="text-embedding-3-small", client=client)
    metrics = [
        Faithfulness(llm=llm),
        AnswerRelevancy(llm=llm, embeddings=emb),
        ContextPrecision(llm=llm),
        ContextRecall(llm=llm),
    ]
    return await ascore_dataset(dataset, metrics, concurrency=8)


df = asyncio.run(main(dataset))
print(df[["faithfulness", "answer_relevancy", "context_precision", "context_recall"]].mean())

How Each Score Is Computed¶

ContextPrecision is rank-aware: the same useful context scores higher at position 1 than at position 3, so it reacts to re-ranking. ContextRecall ignores order and counts only what the reference needs.

LLM Calls per Metric¶

Judge calls dominate cost and latency. k is the number of retrieved contexts. The calls inside one metric run one after another; different rows and metrics run concurrently.

A four-metric RAG suite (Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecall) with k=5 makes 2 + 3 + 5 + 1 = 11 judge calls per row. NoiseSensitivity with the same k adds 13 more, which is why it is usually run on a small slice. Each retry for invalid JSON is an extra billed call (01).

Reading the Scores¶

Scores are relative measurements, not probabilities. Compare runs on the same dataset with the same judge and prompts; take thresholds from a measured baseline minus a tolerance (05) rather than from a number in a blog post. Per-sample scores are noisy (one flipped verdict on a 4-statement answer moves Faithfulness by 0.25), so gate on means and on a floor for catastrophic samples, not on every row.

When Each Metric Misleads¶

Override a prompt to adapt a metric to your domain; the metric keeps its scoring code:

from ragas.metrics.collections import ContextRecall
from ragas.metrics.collections.context_recall.util import ContextRecallPrompt


class StrictRecallPrompt(ContextRecallPrompt):
    instruction = ContextRecallPrompt.instruction + (
        "\nTreat numbers and dates as exact: a mismatch means the statement is not attributed."
    )


metric = ContextRecall(llm=llm)
metric.prompt = StrictRecallPrompt()

Faithfulness keeps two prompts: statement_generator_prompt and nli_statement_prompt. Prompt classes live in each metric's util module with their *Input / *Output models.

Retrieval-Only Metrics Without an LLM¶

These legacy metrics compare retrieved passages with gold passages and cost nothing per run, so they suit large retriever sweeps and unit tests. They need reference_contexts (or the ID lists) in the sample, which test set generation provides (04).

import warnings
warnings.simplefilter("ignore", DeprecationWarning)         # ragas.metrics imports are deprecated

from ragas import SingleTurnSample
from ragas.metrics import (
    IDBasedContextPrecision, IDBasedContextRecall,
    NonLLMContextPrecisionWithReference, NonLLMContextRecall,
)

sample = SingleTurnSample(
    retrieved_contexts=["Paris is the capital of France.", "Berlin is in Germany.", "Eiffel Tower is in Paris"],
    reference_contexts=["Paris is the capital of France.", "Eiffel Tower is in Paris."],
    retrieved_context_ids=["a", "b", "c"],
    reference_context_ids=["a", "c"],
)
for metric in (NonLLMContextPrecisionWithReference(), NonLLMContextRecall(),
               IDBasedContextPrecision(), IDBasedContextRecall()):
    print(metric.name, await metric.single_turn_ascore(sample))      # inside an async function

The legacy classes also run through evaluate() (01). String similarity needs rapidfuzz. ID matching is the most stable retrieval check when your corpus has stable chunk IDs; string similarity helps when chunks are re-split between runs.

Common Mistakes¶

See also¶

• Ragas — Evaluation for LLM, RAG and Agent Applications
• Ragas — Datasets, Samples & Setup
• Ragas — Other Metrics, Agents & Custom
• DeepEval — RAG Metrics
• DeepEval — LLM Testing Guide
• Eval Harness — Tools, Testing & CI