Skip to content
Quantoros

What's the fundamental difference in how an RNN and a transformer read a sequence?

LLM & AILevel: FoundationTransformers & Architecture

What's the fundamental difference in how an RNN and a transformer read a sequence?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

An RNN updates a recurrent state in sequence: the state at position t depends on the previous state and the current input. A transformer uses attention to mix information across permitted positions, without that same per-layer recurrent dependency.

Permitted is important: an encoder can use bidirectional attention, whereas a causal decoder masks future positions. With known training tokens, a causal transformer can compute many positions in parallel while respecting the mask. During ordinary autoregressive generation, however, a new token still depends on previously generated tokens. Transformers do not generate an entire answer simultaneously merely because their training computation is parallel.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial

Related questions