Why is subword tokenization preferred over word-level tokenization?
LLM & AILevel: FoundationTokenization & Embeddings
Why is subword tokenization preferred over word-level tokenization?
Hints
- Separate raw text, a token identifier and its vector representation.
- Consider unseen strings, context length and vocabulary effects across languages.
Answer
Word-level problems: huge vocabulary (millions of words); unknown words become [UNK]; can't handle morphological variants.
Subword benefits:
- Handles out-of-vocabulary words by decomposing them into known subwords
- Manageable vocabulary size (32K–128K typically)
- Learns morphology naturally (prefixes, suffixes)
- Works across all languages with the same tokenizer
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0