Skip to content
Quantoros

Explain the difference between L2 regularization and weight decay in the context of Adam optimizer.

Deep LearningLevel: FoundationNeural networks

Explain the difference between L2 regularization and weight decay in the context of Adam optimizer.

Hints
  • Start from the loss: what changes when a parameter penalty is added?
  • Compare the constraint geometry and behavior near zero. Separately consider feature scaling.

Answer

In plain SGD, L2 regularization (adding λ‖w‖² to the loss) and weight decay (directly shrinking weights by a factor each step) are mathematically equivalent. But with adaptive optimizers like Adam, they're not equivalent — L2 regularization gets folded into the gradient and then scaled by Adam's adaptive per-parameter learning rates, which weakens its regularizing effect inconsistently across parameters. AdamW decouples weight decay from the gradient-based update, applying it directly to the weights, which was shown empirically to generalize better than naive L2 regularization with Adam.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions