Which optimization techniques for training neural nets do you know?
Deep LearningLevel: IntermediateOptimization in neural networks
Which optimization techniques for training neural nets do you know?
Hints
- Trace data and gradients through the model, noting shapes.
- Separate training from inference and identify a necessary assumption.
Answer
- Gradient Descent
- Stochastic Gradient Descent
- Mini-Batch Gradient Descent(best among gradient descents)
- Nesterov Accelerated Gradient
- Momentum
- Adagrad
- AdaDelta
- Adam(best one. less time, more efficient)
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0