What’s the difference between L2 and L1 regularization?
Classic MLLevel: IntermediateRegularization
What’s the difference between L2 and L1 regularization?
Hints
- Start from the loss: what changes when a parameter penalty is added?
- Compare the constraint geometry and behavior near zero. Separately consider feature scaling.
Answer
- Penalty terms: L1 regularization uses the sum of the absolute values of the weights, while L2 regularization uses the sum of the weights squared.
- Feature selection: L1 performs feature selection by reducing the coefficients of some predictors to 0, while L2 does not.
- Computational efficiency: L2 has an analytical solution, while L1 does not.
- Multicollinearity: L2 addresses multicollinearity by constraining the coefficient norm.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0