Why can a word remain in vectorized text even if a contraction containing it is in the stop list?
Classic MLLevel: IntermediateStop-list preprocessing
Why can a word remain in vectorized text even if a contraction containing it is in the stop list?
Hints
- The stop list is matched after preprocessing.
- Surface strings and resulting tokens can differ.
Answer
The tokenizer may split the listed form into different tokens. Apply the same preprocessing and tokenization to stop words and documents, and inspect the actual token stream.
Source and licence: scikit-learn 1.7.2 — feature extraction · BSD-3-Clause · answer: Quantoros editorial