Skip to content
Quantoros

How do you evaluate retrieval quality in a RAG system?

LLM & AILevel: IntermediateRAG Evaluation

How do you evaluate retrieval quality in a RAG system?

Hints
  • Break the query into candidate retrieval, context selection and generation.
  • How would you locate the stage that lost a fact without using only one end-to-end metric?

Answer

Offline retrieval evaluation:

  1. Build eval dataset: 100–200 (question, ground_truth_answer, relevant_doc_ids) triples
  2. Metrics:
    • Recall@k: fraction of relevant docs in top-k retrieved
    • Precision@k: fraction of top-k retrieved that are relevant
    • MRR (Mean Reciprocal Rank): reciprocal of rank of first relevant doc
    • NDCG: normalized discounted cumulative gain — accounts for rank order

Online retrieval evaluation:

  • Track click-through on citations — do users click cited sources? (proxy for retrieval relevance)
  • Track "no useful results" feedback
  • Monitor answer faithfulness (poor faithfulness → poor retrieval or chunk quality)

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions