How would you decide the right learning rate, and what does a learning rate schedule / warmup do?
Deep LearningLevel: IntermediateNeural networks
How would you decide the right learning rate, and what does a learning rate schedule / warmup do?
Hints
- Trace data and gradients through the model, noting shapes.
- Separate training from inference and identify a necessary assumption.
Answer
Too high a learning rate causes the loss to diverge or oscillate; too low causes painfully slow convergence and risk of getting stuck in poor local minima/plateaus. A common approach is a learning rate range test (increase LR exponentially over a few iterations, plot loss, pick the value just before divergence) combined with schedules like cosine decay or step decay to reduce LR as training progresses. Warmup (starting with a small LR and ramping up over the first few hundred/thousand steps) is especially important for Transformers, since large early updates with unstable gradient statistics (particularly with Adam) can destabilize training before the model has "settled."
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT