Explain the Transformer architecture.
LLM & AILevel: FoundationArchitecture & Core Concepts
Explain the Transformer architecture.
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
The Transformer (Vaswani et al., 2017) processes sequences in parallel using attention rather than recurrence. Modern LLMs use a decoder-only variant:
- Token + Positional Embeddings — encode meaning and position (modern: RoPE instead of learned/sinusoidal)
- Multi-Head Self-Attention — tokens attend to each other; multiple heads capture different relationships
- Feed-Forward Network — two linear layers with activation (modern standard: SwiGLU)
- Residual connections + Normalization — stabilize training (modern: Pre-RMSNorm)
- Causal masking — prevents attending to future tokens during generation
Encoder-only (BERT): bidirectional attention, used for classification/NER.
Encoder-decoder (T5): encoder processes input, decoder generates output — for translation/summarization.
Decoder-only (GPT, LLaMA, Claude): dominant for generative AI.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0