What is word embedding, and how does Word2Vec learn embeddings?
NLPLevel: FoundationNatural language processing
What is word embedding, and how does Word2Vec learn embeddings?
Hints
- What does closeness mean in this representation and metric?
- Check normalization, exact versus approximate search and alignment with the training objective.
Answer
Word embeddings represent words as dense, low-dimensional vectors that capture semantic relationships (words with similar meaning/usage end up close together in vector space), unlike sparse one-hot encodings that carry no notion of similarity. Word2Vec learns these embeddings via a shallow neural network trained on one of two tasks: CBOW (predict a target word from its surrounding context words) or Skip-gram (predict surrounding context words from a target word). The learned weight matrix (mapping words to hidden-layer activations) becomes the embedding — words used in similar contexts end up with similar vectors, famously enabling analogies like king - man + woman ≈ queen.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT