Skip to content
Quantoros

What is tokenization? Explain BPE.

LLM & AILevel: FoundationTokenization & Embeddings

What is tokenization? Explain BPE.

Hints
  • Separate raw text, a token identifier and its vector representation.
  • Consider unseen strings, context length and vocabulary effects across languages.

Answer

Tokenization converts raw text into token IDs. Byte Pair Encoding (BPE):

  1. Initialize vocabulary with individual characters (or bytes)
  2. Count all adjacent symbol pairs in the corpus
  3. Merge the most frequent pair into a new symbol
  4. Repeat until vocabulary reaches target size (e.g. 50K)

Result: common words become single tokens; rare/novel words split into subword pieces ("tokenization" → ["token", "ization"]).

WordPiece (BERT): similar to BPE but uses likelihood maximization for merges.
SentencePiece (T5, LLaMA): works directly on raw bytes, language-agnostic.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions