#Tokenization

Neural networks, an LLM included, are nothing but arithmetic on learned numbers, so text has to become numbers first. The tokenizer cuts text into pieces and gives each piece an ID from a fixed vocabulary. The model never sees letters. GPT-4o's tokenizer, for example, cuts Hello world! into Hello, ·world and ! (the · marks a space), and the model receives only their IDs, the list [13225, 2375, 0].

#Where to cut?

There are three obvious choices. Single characters keep the vocabulary tiny, but a sentence becomes a long string of pieces, and every extra token means more work for the model and more room taken in the context window. Whole words keep sentences short, but the vocabulary would need every word, name and typo that could ever appear. LLMs use the compromise between the two: subwords.

Subword tokens give a vocabulary of a manageable size (50,000 to 200,000 entries) where common words like ·the are a single token and rare words are built from parts. Nothing is ever "unknown": in the worst case a word falls apart into single bytes.

#Bytes underneath

Computers store text as bytes using the UTF-8 encoding. A Latin letter takes one byte, a Cyrillic letter two, a Chinese character three, and an emoji four. That gives a neat starting point: any text in any language is a sequence drawn from just 256 possible byte values.

GPT-2 introduced this byte-level trick, and almost every LLM tokenizer since uses it (the SentencePiece tokenizers of the first Llama models reach the same result with "byte fallback"). The price is that a token is not always a whole character: GPT-4o spends two tokens on the 4-byte cat emoji 🐈, and neither half is a valid character on its own.

#Byte-pair encoding

Most LLM tokenizers are trained with byte-pair encoding (BPE). The algorithm is short enough to fit in a paragraph:

1. Start with a vocabulary of single characters (real tokenizers use the 256 possible bytes).
2. Pre-split the text with a simple rule (words, numbers, punctuation, spaces glued to the next word), so merges never cross those borders.
3. Count every pair of neighboring tokens in a large training text.
4. Merge the most frequent pair into a new token and add it to the vocabulary.
5. Repeat until the vocabulary is as big as you want.

Watch it run on a toy corpus, a tiny collection of training text. Notice how est, ·low and ·wid emerge, because they are the patterns that repeat.

To tokenize new text, the tokenizer replays its merges in the order it learned them. That is why slowest, never seen in training, still splits into sensible parts: ·, s, low, est. The space and the s stay apart because the toy corpus never had a word starting with s. GPT-4o's tokenizer, trained on vastly more text, knows ·slow as one token and cuts the word into just ·slow and est.

#A real tokenizer

Here is the actual tokenizer behind GPT-4o (o200k_base, about 200,000 tokens), running in your browser. Try the samples, or paste your own text.

#Not every language pays the same

A tokenizer's merges are learned from its training text, which is mostly English. English words therefore get long, whole-word tokens, while other languages are cut into smaller pieces. Since models are priced, limited and slowed down per token, the same message costs more in some languages than in others.

The jump from GPT-3 to GPT-4o is dramatic: the Russian sentence drops from 81 tokens to 27, and the Hindi one from 125 to 33. A bigger vocabulary costs something too: every token needs its own row of learned numbers in the embedding table (chapter 3). For multilingual text it pays for itself many times over.

#Why tokenization explains odd behavior

Several famous LLM quirks are tokenizer artifacts, not reasoning failures:

• Spelling and counting letters. strawberry reaches the model as one to three chunks, never as ten letters: ·strawberry mid-sentence, st raw berry at the start of a text (GPT-4o tokenizer). Counting the rs means remembering what is inside each chunk.
• Arithmetic. The GPT-4 and GPT-4o tokenizers cut numbers into chunks of up to three digits, counted from the left. In 1234567 + 89 the model sees 123 456 7 and 89: the ones digits it must add, 7 and 9, sit in chunks of different lengths, so the columns of the sum do not line up.
• Languages. Text in languages that were rare in the tokenizer's training data needs more tokens per word, so it is slower and more expensive to process.
• Leading spaces. ·cat and cat are different tokens. The space belongs to the word after it.

#Summary

• A tokenizer maps text to a list of integer IDs and back.
• BPE learns the vocabulary by repeatedly merging the most frequent pair of tokens.
• It works on UTF-8 bytes, so any text can be encoded, even text it never saw.
• Modern vocabularies hold 100k to 200k tokens; one token is about 4 characters of English, and fewer for most other languages.
• Many odd behaviors, from miscounted letters to clumsy arithmetic and pricier non-English text, come from how the text was cut, not from the model's reasoning.

Next, those integer IDs become something the network can actually compute with: vectors.