Skip to content
Quantoros

What does tokenization do to your text, and why is it necessary?

LLM & AILevel: FoundationTransformers & Architecture

What does tokenization do to your text, and why is it necessary?

Hints
  • Separate raw text, a token identifier and its vector representation.
  • Consider unseen strings, context length and vocabulary effects across languages.

Answer

A model only does math over numbers, so raw letters mean nothing to it. Tokenization bridges that. It splits your text into small pieces, usually subwords, drawn from a fixed vocabulary, then swaps each piece for its integer ID.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions