How do you evaluate retrieval quality in a RAG system?
LLM & AILevel: IntermediateRAG Evaluation
How do you evaluate retrieval quality in a RAG system?
Hints
- Break the query into candidate retrieval, context selection and generation.
- How would you locate the stage that lost a fact without using only one end-to-end metric?
Answer
Offline retrieval evaluation:
- Build eval dataset: 100–200 (question, ground_truth_answer, relevant_doc_ids) triples
- Metrics:
- Recall@k: fraction of relevant docs in top-k retrieved
- Precision@k: fraction of top-k retrieved that are relevant
- MRR (Mean Reciprocal Rank): reciprocal of rank of first relevant doc
- NDCG: normalized discounted cumulative gain — accounts for rank order
Online retrieval evaluation:
- Track click-through on citations — do users click cited sources? (proxy for retrieval relevance)
- Track "no useful results" feedback
- Monitor answer faithfulness (poor faithfulness → poor retrieval or chunk quality)
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0