Skip to content
Quantoros

How do you implement RAG over documents containing text and images?

Computer VisionLevel: IntermediateMultimodal RAG

How do you implement RAG over documents containing text and images?

Hints
  • Identify the prediction unit: image, object, pixel or sequence.
  • How do occlusion, lighting or domain changes affect results and evaluation?

Answer

Standard text RAG ignores figures, charts, and diagrams in PDFs — critical information loss for technical documents.

Approaches:

Option 1 — Extract and describe images separately:

PDF → extract images → GPT-4V generates text description of each image → 
index descriptions in vector DB alongside text chunks

Fast, cheap; loses visual nuance; image descriptions may miss important visual details.

Option 2 — Multi-vector retrieval (ColPali/ColQwen):

PDF pages → render as images → embed page images with vision encoder → 
query → retrieve relevant page images → VLM answers from page images

Preserves original visual information; higher cost per query.

Option 3 — Hybrid:

  • Text-based retrieval for most queries
  • Rerank results; if top results reference figures, also retrieve image embeddings for those figures
  • Pass relevant images to VLM for generation

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions