What is the exploding/vanishing gradient problem specific to vanilla RNNs, and how do LSTMs solve it?
Deep LearningLevel: FoundationNeural networks
What is the exploding/vanishing gradient problem specific to vanilla RNNs, and how do LSTMs solve it?
Hints
- What changes in one update and which information is used?
- Check the sign, step size and behavior on an ill-conditioned surface.
Answer
In vanilla RNNs, the same weight matrix is applied repeatedly across time steps, so gradients during backprop-through-time are products of many Jacobians — this compounds multiplicatively and tends to vanish or explode over long sequences, making it hard to learn long-range dependencies. LSTMs introduce a cell state with an additive update path controlled by gating mechanisms (forget, input, output gates), so gradients can flow through the cell state with much less multiplicative decay — the additive structure is the key architectural fix, similar in spirit to residual connections in CNNs.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT