Skip to content
Quantoros

What is the difference between encoder-only, decoder-only, and encoder-decoder architectures?

LLM & AILevel: FoundationArchitecture & Core Concepts

What is the difference between encoder-only, decoder-only, and encoder-decoder architectures?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

ArchitectureAttentionTraining ObjectiveExamplesBest For
Encoder-onlyBidirectionalMasked LMBERT, RoBERTaClassification, NER, embeddings
Decoder-onlyCausal (unidirectional)Next-token predictionGPT, LLaMA, ClaudeText generation, chat
Encoder-decoderCross-attentionSeq2seqT5, BARTTranslation, summarization

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions