Skip to content
Quantoros

How do you evaluate a RAG system?

LLM & AILevel: IntermediateEvaluation

How do you evaluate a RAG system?

Hints
  • Break the query into candidate retrieval, context selection and generation.
  • How would you locate the stage that lost a fact without using only one end-to-end metric?

Answer

Evaluate retrieval and generation separately, then end-to-end:

Retrieval metrics (offline, against labeled dataset):

  • Recall@K — is the relevant document in the top K results?
  • MRR (Mean Reciprocal Rank) — how highly is the relevant document ranked?
  • Context Precision@K — of the K retrieved chunks, what fraction are actually relevant?

Generation metrics (LLM-as-judge or model-based):

  • Faithfulness — is every claim in the answer supported by the retrieved context? (anti-hallucination metric)
  • Answer Relevance — does the answer actually address the query?
  • Context Recall — does the answer use all the important information in the context?

Framework: RAGAS automates all of the above using an LLM as evaluator. Maintain a golden dataset of (query, expected answer, relevant documents) for regression testing after every change.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions