Skip to content
Quantoros

What is the vanishing/exploding gradient problem, and how do modern architectures address it?

Deep LearningLevel: FoundationNeural networks

What is the vanishing/exploding gradient problem, and how do modern architectures address it?

Hints
  • What changes in one update and which information is used?
  • Check the sign, step size and behavior on an ill-conditioned surface.

Answer

In deep networks, gradients are products of many partial derivatives during backpropagation; if these are consistently < 1 (e.g., with sigmoid/tanh saturating), gradients shrink toward zero in early layers (vanishing), stalling learning. If consistently > 1, gradients grow uncontrollably (exploding), destabilizing training. Fixes include: ReLU-family activations (non-saturating for positive inputs), residual/skip connections (ResNets, which give gradients a direct path), batch/layer normalization, careful weight initialization (Xavier/He), and gradient clipping for exploding gradients.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions