Skip to content
Quantoros

What is RAGAS and what metrics does it provide?

LLM & AILevel: IntermediateRAG Evaluation

What is RAGAS and what metrics does it provide?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

RAGAS (Retrieval-Augmented Generation Assessment) is a framework for evaluating RAG pipelines end-to-end.

Core metrics:

MetricMeasuresHow
FaithfulnessIs the answer grounded in the context?LLM checks if each claim in the answer can be traced to retrieved context
Answer RelevancyDoes the answer address the question?LLM generates questions from the answer; compare to original
Context PrecisionAre retrieved chunks relevant?Fraction of retrieved chunks that are relevant to the question
Context RecallAre all relevant chunks retrieved?Fraction of ground-truth answer components that appear in retrieved context

Composite score: RAGAS score = harmonic mean of all four metrics.

Usage:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision]
)

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions