Skip to content
Quantoros

How do Transformers address the limitations of CNNs and RNNs?

Computer VisionLevel: FoundationArchitecture & Core Concepts

How do Transformers address the limitations of CNNs and RNNs?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

RNNs: process sequentially — gradient vanishes over long sequences; can't parallelize.
CNNs: capture local patterns only; need many layers for long-range dependencies.

Transformers solve both: self-attention gives every token direct access to every other token in O(1) steps (vs O(n) for RNN), and the entire sequence is processed in parallel (vs sequential RNN).

Trade-off: Transformers have O(n²) attention complexity — RNNs are O(n). But modern hardware and Flash Attention make this manageable in practice.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions