What is tokenization? Explain BPE.
LLM & AILevel: FoundationTokenization & Embeddings
What is tokenization? Explain BPE.
Hints
- Separate raw text, a token identifier and its vector representation.
- Consider unseen strings, context length and vocabulary effects across languages.
Answer
Tokenization converts raw text into token IDs. Byte Pair Encoding (BPE):
- Initialize vocabulary with individual characters (or bytes)
- Count all adjacent symbol pairs in the corpus
- Merge the most frequent pair into a new symbol
- Repeat until vocabulary reaches target size (e.g. 50K)
Result: common words become single tokens; rare/novel words split into subword pieces ("tokenization" → ["token", "ization"]).
WordPiece (BERT): similar to BPE but uses likelihood maximization for merges.
SentencePiece (T5, LLaMA): works directly on raw bytes, language-agnostic.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0