What do residual connections and normalization each do to keep a deep transformer trainable?
LLM & AILevel: FoundationTransformers & Architecture
What do residual connections and normalization each do to keep a deep transformer trainable?
Hints
- Trace both signal paths through a residual block: the transformation and the skip connection.
- Separately check activation and gradient scales as network depth increases.
Answer
A residual branch forms y = x + F(x). Its derivative contains an identity contribution in addition to the derivative of F, providing a direct route for information and gradients. This can make deep networks easier to optimize, but does not guarantee that all gradients remain well-scaled or that every signal is preserved unchanged.
Layer normalization computes statistics within an example's feature vector and normalizes activations, followed by learned scaling and shifting. It reduces sensitivity to activation scale and helps optimization. Placement before or after a sublayer matters. Residual paths and normalization work alongside initialization, learning rate, precision and sometimes gradient clipping; neither is a proof against instability.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial