What is the difference between BERT and GPT at an architectural level, and why does it affect what tasks each is…
NLPLevel: FoundationNatural language processing
What is the difference between BERT and GPT at an architectural level, and why does it affect what tasks each is naturally good at?
Hints
- Separate text representation, modeling and evaluation.
- Check ambiguity, rare terms and training versus target text distributions.
Answer
BERT uses only the Transformer encoder, trained with a bidirectional masked language modeling objective (predict a randomly masked token using context from both left and right) — well-suited for understanding tasks (classification, NER, extractive QA) where you have the full input available at once. GPT uses only the Transformer decoder, trained with a left-to-right (causal/autoregressive) language modeling objective (predict the next token given only preceding tokens) — well-suited for generation tasks, since it naturally produces text one token at a time without "seeing the future," which BERT's bidirectional context would otherwise leak during generation.
Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT