What does running multiple attention heads let the model do that a single reader couldn't?
LLM & AILevel: IntermediateTransformers & Architecture
What does running multiple attention heads let the model do that a single reader couldn't?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
Multi-head attention computes several attention operations using different learned query, key and value projections. Each head can compare positions in a different representation subspace; the head outputs are concatenated and projected back into the model representation.
This offers multiple simultaneous ways to aggregate information. It does not mean a single head can express only one semantic relationship, nor that specific heads must specialize as grammar, pronoun or topic detectors. Such behavior is an empirical property of a trained model, and heads may be redundant. The benefit is representational flexibility, subject to the parameter and compute budget, rather than a guaranteed assignment of interpretable roles.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial