Explain the difference between stemming and lemmatization.
NLPLevel: FoundationNatural language processing
Explain the difference between stemming and lemmatization.
Hints
- Separate text representation, modeling and evaluation.
- Check ambiguity, rare terms and training versus target text distributions.
Answer
Stemming crudely chops word endings using heuristic rules (e.g., "running" → "run", but also "universe" → "univers") without understanding grammar — fast but can produce non-words. Lemmatization uses vocabulary and morphological analysis (often with a POS tag) to reduce a word to its true dictionary base form (lemma) — e.g., "better" → "good" — producing valid words but requiring more computation and linguistic resources (dictionaries/POS taggers). Lemmatization is generally preferred when interpretability of the reduced form matters; stemming is fine for quick, resource-constrained preprocessing like search indexing.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT