Tokenization is the process of splitting text into small pieces called tokens. Those tokens, not letters or words, are what a language model actually reads.
A token is usually part of a word. Common words are a single token, rare ones get split into several. Each token maps to a number, and those numbers are what the model works with.
Think of it like this. Think of cutting a sentence into Lego bricks. Common words get their own brick. Unusual words get built from several smaller ones. The model only ever sees bricks, never the original sentence.
How it works
- Normalise. Clean up the text: unicode, casing, accents.
- Pre-tokenize. Split roughly on spaces and punctuation.
- Apply the learned rules. Split those rough pieces into the subword units the model knows.
- Map to numbers. Each token becomes an integer ID from the model's vocabulary.
- Add special tokens. Markers for the start of text, separators, and so on.
Types
- Word level. One token per word. Simple, but the vocabulary is huge and unknown words break it.
- Subword. The common choice. Frequent words stay whole, rare ones split into parts, so nothing is ever unknown.
- Byte Pair Encoding (BPE). Repeatedly merges the most frequent pair of characters. Used by GPT models.
- WordPiece. Similar, but merges the pair that most improves the model's likelihood. Used by BERT.
- SentencePiece. Treats text as raw bytes, so it works on any language without assuming spaces separate words.
- Unigram. Learns a vocabulary that maximises the likelihood of the training text, rather than merging greedily.
- Character level. One token per character. Tiny vocabulary, very long sequences.
Typical values
- Vocabulary size: commonly 30,000 to 100,000 tokens.
- BERT: roughly 30,000 tokens. GPT-4: roughly 100,000.
- Tokens per word in English: commonly 1.3 to 2.0, so a 1,000-word document is usually 1,300 to 2,000 tokens.
- Unknown tokens: should be almost nonexistent with subword tokenization, because anything unfamiliar is built from smaller pieces.
Common mistakes
- "One word equals one token." Rarely true. Budget roughly four characters per token for English.
- "Tokenization is a trivial detail." It affects cost, context limits, and how well the model handles code or other languages.
- "Any tokenizer works with any model." Each model has its own. Mixing them produces nonsense.
