How do you evaluate a RAG system?
LLM & AILevel: IntermediateEvaluation
How do you evaluate a RAG system?
Hints
- Break the query into candidate retrieval, context selection and generation.
- How would you locate the stage that lost a fact without using only one end-to-end metric?
Answer
Evaluate retrieval and generation separately, then end-to-end:
Retrieval metrics (offline, against labeled dataset):
- Recall@K — is the relevant document in the top K results?
- MRR (Mean Reciprocal Rank) — how highly is the relevant document ranked?
- Context Precision@K — of the K retrieved chunks, what fraction are actually relevant?
Generation metrics (LLM-as-judge or model-based):
- Faithfulness — is every claim in the answer supported by the retrieved context? (anti-hallucination metric)
- Answer Relevance — does the answer actually address the query?
- Context Recall — does the answer use all the important information in the context?
Framework: RAGAS automates all of the above using an LLM as evaluator. Maintain a golden dataset of (query, expected answer, relevant documents) for regression testing after every change.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0