What are the key components of the Transformer architecture?
LLM & AILevel: FoundationArchitecture & Core Concepts
What are the key components of the Transformer architecture?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
| Component | Purpose |
|---|---|
| Token embeddings | Convert token IDs to dense vectors |
| Positional encoding (RoPE) | Inject sequence order information |
| Multi-head self-attention | Capture relationships between all token pairs |
| Feed-forward network (SwiGLU) | Position-wise non-linear transformation |
| Residual connections | Enable gradient flow through deep networks |
| Layer normalization (RMSNorm) | Stabilize activations, prevent gradient issues |
| Output projection + softmax | Map final hidden states to vocabulary probabilities |
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0