How do diffusion models work?
Computer VisionLevel: IntermediateGenerative Models
How do diffusion models work?
Hints
- Identify the prediction unit: image, object, pixel or sequence.
- How do occlusion, lighting or domain changes affect results and evaluation?
Answer
Diffusion models (DDPM, Stable Diffusion) generate images by learning to reverse a noise-adding process:
Forward process (training data creation):
Original image x₀ → add Gaussian noise step by step → x₁ → x₂ → ... → xT ≈ pure noise
Reverse process (what the model learns):
Pure noise xT → predict and remove noise step by step → ... → x₁ → x₀ (generated image)
The model (U-Net or Transformer) is trained to predict the noise added at each step, conditioned on a text prompt (via cross-attention with CLIP text embeddings).
Stable Diffusion innovation: latent diffusion — run the diffusion process in a compressed latent space (VAE encoder/decoder), not pixel space. 8–16× faster than pixel-space diffusion.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0