What is the vanishing/exploding gradient problem, and how do modern architectures address it?
Deep LearningLevel: FoundationNeural networks
What is the vanishing/exploding gradient problem, and how do modern architectures address it?
Hints
- What changes in one update and which information is used?
- Check the sign, step size and behavior on an ill-conditioned surface.
Answer
In deep networks, gradients are products of many partial derivatives during backpropagation; if these are consistently < 1 (e.g., with sigmoid/tanh saturating), gradients shrink toward zero in early layers (vanishing), stalling learning. If consistently > 1, gradients grow uncontrollably (exploding), destabilizing training. Fixes include: ReLU-family activations (non-saturating for positive inputs), residual/skip connections (ResNets, which give gradients a direct path), batch/layer normalization, careful weight initialization (Xavier/He), and gradient clipping for exploding gradients.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT