What is multi-head attention? Why use multiple heads?
LLM & AILevel: FoundationAttention Mechanisms
What is multi-head attention? Why use multiple heads?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
Multi-head attention runs h parallel attention operations (heads) with different learned projections, then concatenates and projects the results:
MultiHead(Q,K,V) = Concat(head₁,...,headₕ)·Wᵒ
where headᵢ = Attention(Q·Wᵢᵩ, K·Wᵢᴷ, V·Wᵢᵛ)
Why multiple heads? Each head can specialize in different relationship types — one head tracks syntactic dependencies, another tracks coreferences, another tracks positional proximity. Single-head attention averages over all relationship types simultaneously.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0