Explain the difference between L2 regularization and weight decay in the context of Adam optimizer.
Deep LearningLevel: FoundationNeural networks
Explain the difference between L2 regularization and weight decay in the context of Adam optimizer.
Hints
- Start from the loss: what changes when a parameter penalty is added?
- Compare the constraint geometry and behavior near zero. Separately consider feature scaling.
Answer
In plain SGD, L2 regularization (adding λ‖w‖² to the loss) and weight decay (directly shrinking weights by a factor each step) are mathematically equivalent. But with adaptive optimizers like Adam, they're not equivalent — L2 regularization gets folded into the gradient and then scaled by Adam's adaptive per-parameter learning rates, which weakens its regularizing effect inconsistently across parameters. AdamW decouples weight decay from the gradient-based update, applying it directly to the weights, which was shown empirically to generalize better than naive L2 regularization with Adam.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT