How do you implement RAG over documents containing text and images?
Computer VisionLevel: IntermediateMultimodal RAG
How do you implement RAG over documents containing text and images?
Hints
- Identify the prediction unit: image, object, pixel or sequence.
- How do occlusion, lighting or domain changes affect results and evaluation?
Answer
Standard text RAG ignores figures, charts, and diagrams in PDFs — critical information loss for technical documents.
Approaches:
Option 1 — Extract and describe images separately:
PDF → extract images → GPT-4V generates text description of each image →
index descriptions in vector DB alongside text chunks
Fast, cheap; loses visual nuance; image descriptions may miss important visual details.
Option 2 — Multi-vector retrieval (ColPali/ColQwen):
PDF pages → render as images → embed page images with vision encoder →
query → retrieve relevant page images → VLM answers from page images
Preserves original visual information; higher cost per query.
Option 3 — Hybrid:
- Text-based retrieval for most queries
- Rerank results; if top results reference figures, also retrieve image embeddings for those figures
- Pass relevant images to VLM for generation
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0