What is the difference between encoder-only, decoder-only, and encoder-decoder architectures?
LLM & AILevel: FoundationArchitecture & Core Concepts
What is the difference between encoder-only, decoder-only, and encoder-decoder architectures?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
| Architecture | Attention | Training Objective | Examples | Best For |
|---|---|---|---|---|
| Encoder-only | Bidirectional | Masked LM | BERT, RoBERTa | Classification, NER, embeddings |
| Decoder-only | Causal (unidirectional) | Next-token prediction | GPT, LLaMA, Claude | Text generation, chat |
| Encoder-decoder | Cross-attention | Seq2seq | T5, BART | Translation, summarization |
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0