Skip to content
Quantoros

What is self-attention and how is it computed step by step?

LLM & AILevel: FoundationAttention Mechanisms

What is self-attention and how is it computed step by step?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

  1. Project each token embedding into Q, K, V vectors using learned weight matrices (Wq, Wk, Wv)
  2. Compute attention scores: scores = Q·Kᵀ / √d_k (dot product of queries with all keys, scaled)
  3. Apply causal mask (decoder): set future positions to -∞ before softmax
  4. Softmax over scores → attention weights (non-negative, sum to 1 per row)
  5. Output = weighted sum of values: output = attention_weights · V

The query asks "what am I looking for?" Keys say "what do I contain?" Values say "what do I give you?"

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions