How do we select the right regularization parameters?
Classic MLLevel: FoundationRegularization
How do we select the right regularization parameters?
Hints
- Start from the loss: what changes when a parameter penalty is added?
- Compare the constraint geometry and behavior near zero. Separately consider feature scaling.
Answer
Regularization parameters can be chosen using a grid search, for example https://scikit-learn.org/stable/modules/linear_model.html has one formula for the implementing for regularization, alpha in the formula mentioned can be found by doing a RandomSearch or a GridSearch on a set of values and selecting the alpha which gives the least cross validation or validation error.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0