Skip to content
Quantoros

What is cross-attention and where is it used?

LLM & AILevel: IntermediateAttention Mechanisms

What is cross-attention and where is it used?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

Cross-attention is attention where Q comes from one sequence and K,V come from another. In encoder-decoder Transformers (T5, BART), decoder layers use cross-attention to attend to encoder hidden states — letting the decoder "look at" the full input sequence at each generation step.

In decoder-only models, there is no cross-attention — all attention is self-attention with causal masking.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions