Skip to content
Quantoros

Explain the Transformer architecture.

LLM & AILevel: FoundationArchitecture & Core Concepts

Explain the Transformer architecture.

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

The Transformer (Vaswani et al., 2017) processes sequences in parallel using attention rather than recurrence. Modern LLMs use a decoder-only variant:

  • Token + Positional Embeddings — encode meaning and position (modern: RoPE instead of learned/sinusoidal)
  • Multi-Head Self-Attention — tokens attend to each other; multiple heads capture different relationships
  • Feed-Forward Network — two linear layers with activation (modern standard: SwiGLU)
  • Residual connections + Normalization — stabilize training (modern: Pre-RMSNorm)
  • Causal masking — prevents attending to future tokens during generation

Attention(Q,K,V)=softmax(QKTdk)V\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Encoder-only (BERT): bidirectional attention, used for classification/NER.
Encoder-decoder (T5): encoder processes input, decoder generates output — for translation/summarization.
Decoder-only (GPT, LLaMA, Claude): dominant for generative AI.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions