Skip to content
Quantoros

How to set the learning rate?

Deep LearningLevel: IntermediateOptimization in neural networks

How to set the learning rate?

Hints
  • Trace data and gradients through the model, noting shapes.
  • Separate training from inference and identify a necessary assumption.

Answer

There is no straightforward way of finding an optimum learning rate for a model. It involves a lot of hit and trial. Usually starting with a small values such as 0.01 is a good starting point for setting a learning rate and further tweaking it so that it doesn't overshoot or converge too slowly.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0

Related questions