Skip to content
Quantoros

Your RAG system retrieves relevant documents but answers are still wrong. What's happening?

LLM & AILevel: IntermediateGeneration & Synthesis

Your RAG system retrieves relevant documents but answers are still wrong. What's happening?

Hints
  • Break the query into candidate retrieval, context selection and generation.
  • How would you locate the stage that lost a fact without using only one end-to-end metric?

Answer

This is the retrieval-generation gap. Possible causes:

  1. Retrieved but not used — context too long, important info buried in middle (lost-in-the-middle); model uses parametric knowledge instead. Fix: shorter context, explicit grounding instruction.
  2. Chunk boundary problem — answer spans multiple chunks; each chunk alone is incomplete. Fix: larger chunks, parent-child chunking, overlapping chunks.
  3. Synthesis failure — needs combining information from multiple docs; model summarizes each separately. Fix: explicit synthesis instruction; multi-hop decomposition.
  4. Hallucination despite context — model "confabulates" plausible extensions of the retrieved text. Fix: "Answer ONLY based on the provided context. If the answer is not there, say so."
  5. Wrong retrieval despite good metrics — eval metrics measure the wrong thing. Inspect actual retrieved chunks manually.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions