How does CLIP work?
Computer VisionLevel: IntermediateVision-Language Models
How does CLIP work?
Hints
- Identify the prediction unit: image, object, pixel or sequence.
- How do occlusion, lighting or domain changes affect results and evaluation?
Answer
CLIP (Contrastive Language-Image Pretraining, OpenAI 2021) learns to align image and text representations through contrastive learning on 400M image-text pairs:
Training:
- Image encoder (ViT or ResNet) encodes each image → image embedding
- Text encoder (Transformer) encodes each caption → text embedding
- Contrastive loss: maximize cosine similarity of matching pairs; minimize for non-matching pairs
- Trained on (image, caption) pairs scraped from the internet
Zero-shot classification:
Image of a dog
Text prompts: "a photo of a cat", "a photo of a dog", "a photo of a bird"
→ Find text with highest cosine similarity to image embedding
→ "a photo of a dog" → prediction: dog
Why CLIP is foundational: CLIP embeddings bridge vision and language. Used in: zero-shot image classification, semantic image search, conditioning text-to-image generation (DALL-E, Stable Diffusion).
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0