What is Adam? What’s the main difference between Adam and SGD?
Deep LearningLevel: IntermediateOptimization in neural networks
What is Adam? What’s the main difference between Adam and SGD?
Hints
- Trace data and gradients through the model, noting shapes.
- Separate training from inference and identify a necessary assumption.
Answer
Adam (Adaptive Moment Estimation) is a optimization technique for training neural networks. on an average, it is the best optimizer .It works with momentums of first and second order. The intuition behind the Adam is that we don’t want to roll so fast just because we can jump over the minimum, we want to decrease the velocity a little bit for a careful search.
Adam tends to converge faster, while SGD often converges to more optimal solutions. SGD's high variance disadvantages gets rectified by Adam (as advantage for Adam).
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0