Skip to content
Quantoros

What does running multiple attention heads let the model do that a single reader couldn't?

LLM & AILevel: IntermediateTransformers & Architecture

What does running multiple attention heads let the model do that a single reader couldn't?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

Multi-head attention computes several attention operations using different learned query, key and value projections. Each head can compare positions in a different representation subspace; the head outputs are concatenated and projected back into the model representation.

This offers multiple simultaneous ways to aggregate information. It does not mean a single head can express only one semantic relationship, nor that specific heads must specialize as grammar, pronoun or topic detectors. Such behavior is an empirical property of a trained model, and heads may be redundant. The benefit is representational flexibility, subject to the parameter and compute budget, rather than a guaranteed assignment of interpretable roles.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial

Related questions