What are the trade-offs in using a large vocabulary size?
LLM & AILevel: IntermediateTokenization & Embeddings
What are the trade-offs in using a large vocabulary size?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
| Aspect | Larger Vocabulary | Smaller Vocabulary |
|---|---|---|
| Token fragmentation | Less (rare words stay whole) | More (words split into pieces) |
| Sequence length | Shorter | Longer |
| Memory (embedding layer) | Higher | Lower |
| Softmax compute | Slower | Faster |
| Rare word coverage | Better | Worse |
Typical sweet spot: 32K–128K. LLaMA 3 uses ~128K; GPT-4 uses ~100K.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0