Skip to content
Quantoros

How would you decide the right learning rate, and what does a learning rate schedule / warmup do?

Deep LearningLevel: IntermediateNeural networks

How would you decide the right learning rate, and what does a learning rate schedule / warmup do?

Hints
  • Trace data and gradients through the model, noting shapes.
  • Separate training from inference and identify a necessary assumption.

Answer

Too high a learning rate causes the loss to diverge or oscillate; too low causes painfully slow convergence and risk of getting stuck in poor local minima/plateaus. A common approach is a learning rate range test (increase LR exponentially over a few iterations, plot loss, pick the value just before divergence) combined with schedules like cosine decay or step decay to reduce LR as training progresses. Warmup (starting with a small LR and ramping up over the first few hundred/thousand steps) is especially important for Transformers, since large early updates with unstable gradient statistics (particularly with Adam) can destabilize training before the model has "settled."

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions