Skip to content
Quantoros

What are the trade-offs in using a large vocabulary size?

LLM & AILevel: IntermediateTokenization & Embeddings

What are the trade-offs in using a large vocabulary size?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

AspectLarger VocabularySmaller Vocabulary
Token fragmentationLess (rare words stay whole)More (words split into pieces)
Sequence lengthShorterLonger
Memory (embedding layer)HigherLower
Softmax computeSlowerFaster
Rare word coverageBetterWorse

Typical sweet spot: 32K–128K. LLaMA 3 uses ~128K; GPT-4 uses ~100K.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions