What does tokenization do to your text, and why is it necessary?
LLM & AILevel: FoundationTransformers & Architecture
What does tokenization do to your text, and why is it necessary?
Hints
- Separate raw text, a token identifier and its vector representation.
- Consider unseen strings, context length and vocabulary effects across languages.
Answer
A model only does math over numbers, so raw letters mean nothing to it. Tokenization bridges that. It splits your text into small pieces, usually subwords, drawn from a fixed vocabulary, then swaps each piece for its integer ID.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0