What happens when the learning rate is too large? Too small?
Deep LearningLevel: FoundationOptimization in neural networks
What happens when the learning rate is too large? Too small?
Hints
- Trace data and gradients through the model, noting shapes.
- Separate training from inference and identify a necessary assumption.
Answer
A large learning rate can accelerate the training. However, it is possible that we "shoot" too far and miss the minimum of the function that we want to optimize, which will not result in the best solution. On the other hand, training with a small learning rate takes more time but it is possible to find a more precise minimum. The downside can be that the solution is stuck in a local minimum, and the weights won't update even if it is not the best possible global solution.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0