Skip to content
Quantoros

When do we need to perform feature normalization for linear models? When it’s okay not to do it?

Classic MLLevel: IntermediateRegularization

When do we need to perform feature normalization for linear models? When it’s okay not to do it?

Hints
  • How do feature units affect distances, coefficients and optimization?
  • Compare penalized and unpenalized models; which split may fit the transformation?

Answer

Scale features when their units would distort regularization or make optimization poorly conditioned. L1/L2 penalties act on coefficient magnitudes: expressing one feature in meters versus millimeters changes the coefficient needed for the same prediction, and therefore changes its penalty. Scaling can also improve gradient-based convergence.

Unregularized least squares with an intercept has the same fitted predictions under invertible feature rescaling in exact arithmetic, so normalization is not mathematically mandatory there. Numerical conditioning can still justify it. Fit the scaler only on training data and reuse it for validation and inference; avoid centering sparse matrices if that would densify them.

For ridge, the closed-form penalty enters XᵀX + λI, not X itself.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0 · answer: Quantoros editorial

Related questions