What is self-attention and how is it computed step by step?
LLM & AILevel: FoundationAttention Mechanisms
What is self-attention and how is it computed step by step?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
- Project each token embedding into Q, K, V vectors using learned weight matrices (Wq, Wk, Wv)
- Compute attention scores:
scores = Q·Kᵀ / √d_k(dot product of queries with all keys, scaled) - Apply causal mask (decoder): set future positions to -∞ before softmax
- Softmax over scores → attention weights (non-negative, sum to 1 per row)
- Output = weighted sum of values:
output = attention_weights · V
The query asks "what am I looking for?" Keys say "what do I contain?" Values say "what do I give you?"
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0