Skip to content
Quantoros

Explain the difference between stemming and lemmatization.

NLPLevel: FoundationNatural language processing

Explain the difference between stemming and lemmatization.

Hints
  • Separate text representation, modeling and evaluation.
  • Check ambiguity, rare terms and training versus target text distributions.

Answer

Stemming crudely chops word endings using heuristic rules (e.g., "running" → "run", but also "universe" → "univers") without understanding grammar — fast but can produce non-words. Lemmatization uses vocabulary and morphological analysis (often with a POS tag) to reduce a word to its true dictionary base form (lemma) — e.g., "better" → "good" — producing valid words but requiring more computation and linguistic resources (dictionaries/POS taggers). Lemmatization is generally preferred when interpretability of the reduced form matters; stemming is fine for quick, resource-constrained preprocessing like search indexing.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions