Skip to content
Quantoros

What does CountVectorizer.transform do with a document containing only tokens absent from its fitted vocabulary?

Classic MLLevel: IntermediateVocabulary outside training

What does CountVectorizer.transform do with a document containing only tokens absent from its fitted vocabulary?

Hints
  • Vocabulary learning happened at fit.
  • A zero row still has the fitted number of columns.

Answer

A zero feature row for that document. It does not extend the vocabulary during transform; monitor out-of-vocabulary rates because the classifier then sees almost no text evidence.

Source and licence: scikit-learn 1.7.2 — feature extraction · BSD-3-Clause · answer: Quantoros editorial

Related questions