What is a Large Language Model (LLM) and how does it work?
LLM & AILevel: FoundationArchitecture & Core Concepts
What is a Large Language Model (LLM) and how does it work?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
An LLM is a neural network — primarily a Transformer decoder — trained on massive text corpora to predict the next token. Key mechanisms:
- Tokenization — text split into subword tokens via BPE/WordPiece
- Embeddings — tokens mapped to dense vectors via a lookup table
- Self-Attention — each token attends to all previous tokens to build contextual representations
- Feed-Forward layers — non-linear transformations applied token-wise
- Autoregressive generation — outputs one token at a time, conditioning on all previous tokens
Training uses next-token prediction (cross-entropy loss) over trillions of tokens, followed by SFT (instruction tuning) and RLHF/DPO for alignment.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0