Skip to content
Quantoros

What regularization techniques for neural nets do you know?

Deep LearningLevel: IntermediateNeural networks

What regularization techniques for neural nets do you know?

Hints
  • Start from the loss: what changes when a parameter penalty is added?
  • Compare the constraint geometry and behavior near zero. Separately consider feature scaling.

Answer

  • L1 Regularization - Defined as the sum of absolute values of the individual parameters. The L1 penalty causes a subset of the weights to become zero, suggesting that the corresponding features may safely be discarded.
  • L2 Regularization - Defined as the sum of square of individual parameters. Often supported by regularization hyperparameter alpha. It results in weight decay.
  • Data Augmentation - This requires some fake data to be created as a part of training set.
  • Drop Out : This is most effective regularization technique for neural nets. Few random nodes in each layer is deactivated in forward pass. This allows the algorithm to train on different set of nodes in each iterations.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0

Related questions