Skip to content
Quantoros

What are embeddings and what do they capture?

LLM & AILevel: FoundationTokenization & Embeddings

What are embeddings and what do they capture?

Hints
  • What does closeness mean in this representation and metric?
  • Check normalization, exact versus approximate search and alignment with the training objective.

Answer

An embedding is a dense, fixed-size vector representation of a discrete token. The embedding matrix (shape: vocab_size × d_model) maps each token ID to a vector through a simple lookup.

What embeddings capture through training:

  • Semantic similarity — synonyms have nearby vectors
  • Analogy relationships — king - man + woman ≈ queen
  • Syntactic roles — verbs cluster together
  • Contextual meaning — in Transformers, the embedding is further refined by each attention layer, so the final hidden state is a contextualized embedding

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions