What’s the learning rate?
Deep LearningLevel: FoundationOptimization in neural networks
What’s the learning rate?
Hints
- Trace data and gradients through the model, noting shapes.
- Separate training from inference and identify a necessary assumption.
Answer
The learning rate is an important hyperparameter that controls how quickly the model is adapted to the problem during the training. It can be seen as the "step width" during the parameter updates, i.e. how far the weights are moved into the direction of the minimum of our optimization problem.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0