What is TF-IDF, and what problem does it solve compared to raw word counts?
NLPLevel: FoundationNatural language processing
What is TF-IDF, and what problem does it solve compared to raw word counts?
Hints
- Separate text representation, modeling and evaluation.
- Check ambiguity, rare terms and training versus target text distributions.
Answer
TF-IDF (Term Frequency–Inverse Document Frequency) weights a word's importance in a document by how often it appears there (TF) discounted by how common it is across all documents (IDF = log(N / documents containing the term)). This solves the problem where raw word counts overweight extremely common words ("the", "is") that appear everywhere and carry little distinguishing information, while giving more weight to words that are frequent in a specific document but rare across the corpus (which tend to be more topically meaningful).
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT