Skip to content
Quantoros

How does CLIP work?

Computer VisionLevel: IntermediateVision-Language Models

How does CLIP work?

Hints
  • Identify the prediction unit: image, object, pixel or sequence.
  • How do occlusion, lighting or domain changes affect results and evaluation?

Answer

CLIP (Contrastive Language-Image Pretraining, OpenAI 2021) learns to align image and text representations through contrastive learning on 400M image-text pairs:

Training:

  • Image encoder (ViT or ResNet) encodes each image → image embedding
  • Text encoder (Transformer) encodes each caption → text embedding
  • Contrastive loss: maximize cosine similarity of matching pairs; minimize for non-matching pairs
  • Trained on (image, caption) pairs scraped from the internet

Zero-shot classification:

Image of a dog
Text prompts: "a photo of a cat", "a photo of a dog", "a photo of a bird"
→ Find text with highest cosine similarity to image embedding
→ "a photo of a dog" → prediction: dog

Why CLIP is foundational: CLIP embeddings bridge vision and language. Used in: zero-shot image classification, semantic image search, conditioning text-to-image generation (DALL-E, Stable Diffusion).

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions