Your RAG system retrieves relevant documents but answers are still wrong. What's happening?
LLM & AILevel: IntermediateGeneration & Synthesis
Your RAG system retrieves relevant documents but answers are still wrong. What's happening?
Hints
- Break the query into candidate retrieval, context selection and generation.
- How would you locate the stage that lost a fact without using only one end-to-end metric?
Answer
This is the retrieval-generation gap. Possible causes:
- Retrieved but not used — context too long, important info buried in middle (lost-in-the-middle); model uses parametric knowledge instead. Fix: shorter context, explicit grounding instruction.
- Chunk boundary problem — answer spans multiple chunks; each chunk alone is incomplete. Fix: larger chunks, parent-child chunking, overlapping chunks.
- Synthesis failure — needs combining information from multiple docs; model summarizes each separately. Fix: explicit synthesis instruction; multi-hop decomposition.
- Hallucination despite context — model "confabulates" plausible extensions of the retrieved text. Fix: "Answer ONLY based on the provided context. If the answer is not there, say so."
- Wrong retrieval despite good metrics — eval metrics measure the wrong thing. Inspect actual retrieved chunks manually.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0