Skip to content
Quantoros

What is multi-head attention? Why use multiple heads?

LLM & AILevel: FoundationAttention Mechanisms

What is multi-head attention? Why use multiple heads?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

Multi-head attention runs h parallel attention operations (heads) with different learned projections, then concatenates and projects the results:

MultiHead(Q,K,V) = Concat(head₁,...,headₕ)·Wᵒ
where headᵢ = Attention(Q·Wᵢᵩ, K·Wᵢᴷ, V·Wᵢᵛ)

Why multiple heads? Each head can specialize in different relationship types — one head tracks syntactic dependencies, another tracks coreferences, another tracks positional proximity. Single-head attention averages over all relationship types simultaneously.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions