Why do categorical variables need preprocessing in scikit-learn, compared to other tools?
Classic MLLevel: IntermediateCategorical data interfaces
Why do categorical variables need preprocessing in scikit-learn, compared to other tools?
Hints
- State the task, inputs and prediction target.
- Compare with a simple baseline: assumptions, objective and independent validation.
Answer
Most estimators expect homogeneous numeric arrays or sparse matrices rather than a categorical variable type, so categories need a suitable encoding. Integer category IDs are not necessarily meaningful numerical magnitudes. ColumnTransformer can apply different transformations to different columns. HistGradientBoosting estimators are an important exception: they can recognize supported dataframe categorical dtypes through categorical_features='from_dtype'. The correct preprocessing depends on the estimator, not on a universal rule to one-hot encode everything.
Source and licence: scikit-learn developers — Official FAQ · BSD-3-Clause · answer: Quantoros editorial