How to set the learning rate?
Deep LearningLevel: IntermediateOptimization in neural networks
How to set the learning rate?
Hints
- Trace data and gradients through the model, noting shapes.
- Separate training from inference and identify a necessary assumption.
Answer
There is no straightforward way of finding an optimum learning rate for a model. It involves a lot of hit and trial. Usually starting with a small values such as 0.01 is a good starting point for setting a learning rate and further tweaking it so that it doesn't overshoot or converge too slowly.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0