How do LLaVA and GPT-4V connect vision to language models?
Computer VisionLevel: IntermediateVision-Language Models
How do LLaVA and GPT-4V connect vision to language models?
Hints
- Identify the prediction unit: image, object, pixel or sequence.
- How do occlusion, lighting or domain changes affect results and evaluation?
Answer
LLaVA (Large Language and Vision Assistant) architecture:
- Visual encoder (CLIP ViT-L) processes the image → visual tokens
- Projection layer (MLP) maps visual tokens to language embedding space
- LLM (LLaMA/Mistral) processes concatenation of visual tokens + text tokens
Training:
- Pre-training: only projection layer; train to align image features with language space on image-caption data
- Instruction fine-tuning: full model fine-tuned on visual instruction following (GPT-4-generated VQA data)
GPT-4V (and GPT-4o): uses similar architecture with proprietary vision encoder + multimodal pre-training at much larger scale. GPT-4o additionally processes audio natively in the same model.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0