What does CountVectorizer.transform do with a document containing only tokens absent from its fitted vocabulary?
Classic MLLevel: IntermediateVocabulary outside training
What does CountVectorizer.transform do with a document containing only tokens absent from its fitted vocabulary?
Hints
- Vocabulary learning happened at fit.
- A zero row still has the fitted number of columns.
Answer
A zero feature row for that document. It does not extend the vocabulary during transform; monitor out-of-vocabulary rates because the classifier then sees almost no text evidence.
Source and licence: scikit-learn 1.7.2 — feature extraction · BSD-3-Clause · answer: Quantoros editorial