Skip to content
Quantoros

What is TF-IDF? How is it useful for text classification?

NLPLevel: IntermediateText classification

What is TF-IDF? How is it useful for text classification?

Hints
  • Separate text representation, modeling and evaluation.
  • Check ambiguity, rare terms and training versus target text distributions.

Answer

Term Frequency (TF) is a scoring of the frequency of the word in the current document. Inverse Document Frequency(IDF) is a scoring of how rare the word is across documents. It is used in scenario where highly recurring words may not contain as much informational content as the domain specific words. For example, words like “the” that are frequent across all documents therefore need to be less weighted. The TF-IDF score highlights words that are distinct (contain useful information) in a given document.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0

Related questions