What's the fundamental difference in how an RNN and a transformer read a sequence?
LLM & AILevel: FoundationTransformers & Architecture
What's the fundamental difference in how an RNN and a transformer read a sequence?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
An RNN updates a recurrent state in sequence: the state at position t depends on the previous state and the current input. A transformer uses attention to mix information across permitted positions, without that same per-layer recurrent dependency.
Permitted is important: an encoder can use bidirectional attention, whereas a causal decoder masks future positions. With known training tokens, a causal transformer can compute many positions in parallel while respecting the mask. During ordinary autoregressive generation, however, a new token still depends on previously generated tokens. Transformers do not generate an entire answer simultaneously merely because their training computation is parallel.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial