What are embeddings and what do they capture?
LLM & AILevel: FoundationTokenization & Embeddings
What are embeddings and what do they capture?
Hints
- What does closeness mean in this representation and metric?
- Check normalization, exact versus approximate search and alignment with the training objective.
Answer
An embedding is a dense, fixed-size vector representation of a discrete token. The embedding matrix (shape: vocab_size × d_model) maps each token ID to a vector through a simple lookup.
What embeddings capture through training:
- Semantic similarity — synonyms have nearby vectors
- Analogy relationships — king - man + woman ≈ queen
- Syntactic roles — verbs cluster together
- Contextual meaning — in Transformers, the embedding is further refined by each attention layer, so the final hidden state is a contextualized embedding
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0