What are N-grams? How can we use them?
NLPLevel: IntermediateText classification
What are N-grams? How can we use them?
Hints
- Separate text representation, modeling and evaluation.
- Check ambiguity, rare terms and training versus target text distributions.
Answer
The function to tokenize into consecutive sequences of words is called n-grams. It can be used to find out N most co-occurring words (how often word X is followed by word Y) in a given sentence.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0