Skip to content
Quantoros

What are the key components of the Transformer architecture?

LLM & AILevel: FoundationArchitecture & Core Concepts

What are the key components of the Transformer architecture?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

ComponentPurpose
Token embeddingsConvert token IDs to dense vectors
Positional encoding (RoPE)Inject sequence order information
Multi-head self-attentionCapture relationships between all token pairs
Feed-forward network (SwiGLU)Position-wise non-linear transformation
Residual connectionsEnable gradient flow through deep networks
Layer normalization (RMSNorm)Stabilize activations, prevent gradient issues
Output projection + softmaxMap final hidden states to vocabulary probabilities

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions