What is cross-attention and where is it used?
LLM & AILevel: IntermediateAttention Mechanisms
What is cross-attention and where is it used?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
Cross-attention is attention where Q comes from one sequence and K,V come from another. In encoder-decoder Transformers (T5, BART), decoder layers use cross-attention to attend to encoder hidden states — letting the decoder "look at" the full input sequence at each generation step.
In decoder-only models, there is no cross-attention — all attention is self-attention with causal masking.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0