Skip to content
Quantoros

What is the exploding/vanishing gradient problem specific to vanilla RNNs, and how do LSTMs solve it?

Deep LearningLevel: FoundationNeural networks

What is the exploding/vanishing gradient problem specific to vanilla RNNs, and how do LSTMs solve it?

Hints
  • What changes in one update and which information is used?
  • Check the sign, step size and behavior on an ill-conditioned surface.

Answer

In vanilla RNNs, the same weight matrix is applied repeatedly across time steps, so gradients during backprop-through-time are products of many Jacobians — this compounds multiplicatively and tends to vanish or explode over long sequences, making it hard to learn long-range dependencies. LSTMs introduce a cell state with an additive update path controlled by gating mechanisms (forget, input, output gates), so gradients can flow through the cell state with much less multiplicative decay — the additive structure is the key architectural fix, similar in spirit to residual connections in CNNs.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions