Skip to content
Quantoros

Why is subword tokenization preferred over word-level tokenization?

LLM & AILevel: FoundationTokenization & Embeddings

Why is subword tokenization preferred over word-level tokenization?

Hints
  • Separate raw text, a token identifier and its vector representation.
  • Consider unseen strings, context length and vocabulary effects across languages.

Answer

Word-level problems: huge vocabulary (millions of words); unknown words become [UNK]; can't handle morphological variants.

Subword benefits:

  • Handles out-of-vocabulary words by decomposing them into known subwords
  • Manageable vocabulary size (32K–128K typically)
  • Learns morphology naturally (prefixes, suffixes)
  • Works across all languages with the same tokenizer

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions