How do Transformers address the limitations of CNNs and RNNs?
Computer VisionLevel: FoundationArchitecture & Core Concepts
How do Transformers address the limitations of CNNs and RNNs?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
RNNs: process sequentially — gradient vanishes over long sequences; can't parallelize.
CNNs: capture local patterns only; need many layers for long-range dependencies.
Transformers solve both: self-attention gives every token direct access to every other token in O(1) steps (vs O(n) for RNN), and the entire sequence is processed in parallel (vs sequential RNN).
Trade-off: Transformers have O(n²) attention complexity — RNNs are O(n). But modern hardware and Flash Attention make this manageable in practice.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0