• October 25, 2026
• 7 min read

Ragas — Evaluation for LLM, RAG and Agent Applications¶

Ragas (ragas on PyPI, GitHub vibrantlabsai/ragas, formerly explodinggradients/ragas) is an Apache-2.0 Python library that scores LLM applications. It supplies three things: a catalogue of metrics (LLM-judged, embedding-based and pure string/NLP), synthetic test set generation from your own documents, and a small experiment runner that stores results next to the dataset. It does not ship a test runner or a dashboard: you call it from pytest, a script or CI and decide what a failing score means.

This guide covers ragas 0.4.3 (released 13 January 2026), the latest release on PyPI.

flowchart LR
  D["Dataset<br/>user_input, response,<br/>retrieved_contexts, reference"] --> M
  subgraph M["Metric.ascore(...)"]
    direction TB
    J["judge LLM<br/>llm_factory(client)"]
    E["embeddings<br/>embedding_factory(client)"]
    P["prompt + JSON schema<br/>→ structured output"]
    J --> P
    E --> P
  end
  M --> S["MetricResult<br/>value, reason"]
  S --> T["table / experiment CSV<br/>per-sample scores"]
  T --> G["CI gate<br/>mean, floor, baseline"]

Installation¶

uv add ragas "langchain-community<0.4.2"
uv add pandas sacrebleu rouge-score rapidfuzz     # to_pandas() and the NLP / string metrics
uv add "datacompy<1"                              # only for DataCompyScore
uv add "ragas[git]"                               # only for version_experiment()
uv add --dev pytest

Python 3.9 or newer. ragas pulls in langchain, langchain-community, langchain-openai, openai, instructor, datasets and diskcache as hard dependencies, so the install is large.

Provider SDKs other than openai are not dependencies: add anthropic or litellm when you need them (see 01 for provider caveats).

Two APIs in One Package¶

0.4 introduced a new metric system without removing the old one. Tutorials mix both, so identify which you are reading before copying code.

evaluate() accepts only legacy metric objects; passing a collections metric raises TypeError: All metrics must be initialised metric objects. The guide shows the current API first and calls out the legacy path where it is the only option.

Section Map¶

When to Use Ragas vs DeepEval vs promptfoo vs Custom¶

Ragas and DeepEval overlap on RAG metrics (faithfulness, answer relevancy, context precision and recall) but differ in implementation, prompts and score scales, so numbers from one are not comparable with the other. DeepEval also wraps four Ragas metrics as RagasMetric (see DeepEval RAG metrics). Pick one library per metric family and keep it fixed across runs.

Minimal Example¶

import asyncio

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextRecall, Faithfulness


async def main() -> None:
    client = AsyncOpenAI()                                   # reads OPENAI_API_KEY
    llm = llm_factory("gpt-4o-mini", client=client)          # judge model; async client required

    sample = dict(
        user_input="What is the refund window?",
        response="Refunds are accepted within 30 days.",
        retrieved_contexts=["Our policy: refunds are accepted within 30 days of purchase."],
        reference="Refunds are accepted within 30 days of purchase.",
    )
    faithfulness = await Faithfulness(llm=llm).ascore(
        user_input=sample["user_input"],
        response=sample["response"],
        retrieved_contexts=sample["retrieved_contexts"],
    )
    recall = await ContextRecall(llm=llm).ascore(
        user_input=sample["user_input"],
        retrieved_contexts=sample["retrieved_contexts"],
        reference=sample["reference"],
    )
    print(faithfulness.value, recall.value)                  # floats in 0.0 - 1.0


asyncio.run(main())

Each metric takes only the fields it needs as keyword arguments and returns a MetricResult. MetricResult behaves like its value in comparisons and arithmetic and also exposes .value, .reason and .traces. See 02 for which fields each metric requires and 05 for gating on the scores.

Names from Older Tutorials¶

Quick Rules¶

1. Decide the API first — new code uses ragas.metrics.collections with llm_factory; use legacy classes only for metrics that have no collections version.
2. Pin langchain-community<0.4.2 and ragas==0.4.3 — the library is in transition and renames happen between minors.
3. Use an async client (AsyncOpenAI, litellm.acompletion) and create it inside the running event loop; reusing one client across several asyncio.run() calls fails intermittently.
4. Bound concurrency yourself — abatch_score and @experiment start every row at once; wrap calls in an asyncio.Semaphore.
5. Treat NaN as a failure, not as zero — a failed judge call must fail the gate, not vanish from the mean.
6. Fix the judge — one judge model, temperature 0, same metric versions; a new judge needs a new baseline.
7. Keep a small golden set and a cache — 20 to 50 reviewed rows scored with DiskCacheBackend keeps CI cheap and stable.
8. Test your own wiring offline — a scripted InstructorBaseRagasLLM or a local fake server proves your harness and gates without spending tokens.

See also¶

• Digital Garden: Knowledge Base
• Python Libraries
• DeepEval — LLM Testing Guide
• DeepEval — RAG Metrics
• promptfoo — LLM Evals & Red Teaming from YAML
• Eval Harness — Tools, Testing & CI
• Pytest — Python Testing Framework
• LangChain — LLM Application Framework
• LiteLLM — One API for 100+ LLM Providers
• Agentic AI Architecture