Skip to content
Quantoros

How do LLaVA and GPT-4V connect vision to language models?

Computer VisionLevel: IntermediateVision-Language Models

How do LLaVA and GPT-4V connect vision to language models?

Hints
  • Identify the prediction unit: image, object, pixel or sequence.
  • How do occlusion, lighting or domain changes affect results and evaluation?

Answer

LLaVA (Large Language and Vision Assistant) architecture:

  1. Visual encoder (CLIP ViT-L) processes the image → visual tokens
  2. Projection layer (MLP) maps visual tokens to language embedding space
  3. LLM (LLaMA/Mistral) processes concatenation of visual tokens + text tokens

Training:

  1. Pre-training: only projection layer; train to align image features with language space on image-caption data
  2. Instruction fine-tuning: full model fine-tuned on visual instruction following (GPT-4-generated VQA data)

GPT-4V (and GPT-4o): uses similar architecture with proprietary vision encoder + multimodal pre-training at much larger scale. GPT-4o additionally processes audio natively in the same model.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions