Skip to content
Quantoros

Why do categorical variables need preprocessing in scikit-learn, compared to other tools?

Classic MLLevel: IntermediateCategorical data interfaces

Why do categorical variables need preprocessing in scikit-learn, compared to other tools?

Hints
  • State the task, inputs and prediction target.
  • Compare with a simple baseline: assumptions, objective and independent validation.

Answer

Most estimators expect homogeneous numeric arrays or sparse matrices rather than a categorical variable type, so categories need a suitable encoding. Integer category IDs are not necessarily meaningful numerical magnitudes. ColumnTransformer can apply different transformations to different columns. HistGradientBoosting estimators are an important exception: they can recognize supported dataframe categorical dtypes through categorical_features='from_dtype'. The correct preprocessing depends on the estimator, not on a universal rule to one-hot encode everything.

Source and licence: scikit-learn developers — Official FAQ · BSD-3-Clause · answer: Quantoros editorial

Related questions