Skip to content
Quantoros

Your RAG system returns duplicate or redundant results. How do you fix it?

LLM & AILevel: IntermediateTroubleshooting Scenarios

Your RAG system returns duplicate or redundant results. How do you fix it?

Hints
  • Break the query into candidate retrieval, context selection and generation.
  • How would you locate the stage that lost a fact without using only one end-to-end metric?

Answer

  1. MMR (Maximal Marginal Relevance): when selecting top-K, prefer results that are both relevant AND dissimilar to already-selected results
  2. Deduplication at indexing: detect near-duplicate chunks during ingestion (cosine similarity > 0.98) and deduplicate before indexing
  3. Parent-child chunking: since child chunks from the same parent would be retrieved together, deduplicate to one parent per result
  4. Post-retrieval deduplication: cluster retrieved chunks by similarity; keep one representative per cluster

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions