#Vectors and embeddings

After tokenization, the text is a list of integers: The cat sat becomes [976, 9059, 10139]. A neural network can only add and multiply numbers, so the obvious idea is to feed it those IDs directly. That does not work, because an ID is only a position in the vocabulary list, decided by the order in which the tokenizer happened to learn its pieces. Here are real IDs from the GPT-4o tokenizer:

By these numbers, "catalog" is about nine times closer to "cat" than "kitten" is. Any arithmetic the network did on IDs would be arithmetic on accidents of numbering, and nothing it learned about "cat" would carry over to "kitten".

What the model needs instead is a description of each token in which similar tokens get similar numbers. One number cannot do that: a kitten is alive, an animal, a cat, young, small, and much more. So every token gets a whole list of numbers, a vector, with room for many aspects at once.

#What the numbers mean

Imagine filling in a character sheet for every word: is it alive? an animal? a cat? young? royal? female? a machine? One number answers each question. Words with similar meanings get similar sheets: "cat" and "kitten" differ only in the "young" row, while "car" shares almost nothing with either.

Real models use the same idea with one important twist: nobody chooses the questions. The numbers start out random, and training adjusts them step by step, keeping whatever helps predict the next token. Tokens used in similar ways end up with similar vectors, because that makes the prediction easier. The result looks like the learned view in the figure.

That view is the same sheet after a random turn of the whole space: turning mixes every number into every other, but leaves the angles between the vectors alone, so every similarity survives. The information is there, just spread across all the numbers at once. A single number, say number 1,234 of the vector for ·cat, usually means nothing you could put a name to. What carries meaning is the whole pattern: the direction the vector points in. (Researchers can still find directions with readable meanings, as chapter 6 shows.)

#Lists of numbers as arrows

Why talk about a list of numbers "pointing" somewhere? Because a vector, an ordered list of numbers such as $(0.8, -1.2, 0.3)$, can be drawn. With two numbers it is an arrow on paper, from the origin to the point (x, y); with three, an arrow in a room. Two lists with similar numbers are two arrows pointing almost the same way.

Language models use far more than three numbers:

The number with a B in a model's name is its parameter count in billions: Llama 3 8B has about 8 billion.

You cannot picture 4,096 dimensions, but the geometry still works the same way: vectors have lengths, they point in directions, and the angle between two vectors tells you how alike they are.

#The embedding table

So the model needs one vector for every token in its vocabulary. It stores them all in one big matrix, the embedding table, with one row per token. Turning a token into a vector is nothing more than reading its row: the ID is still useful, but only as an address that says which row to read, not as a meaning.

The table for a 200,000-token vocabulary with 4,096 dimensions holds 819 million numbers, a big share of a small model's parameters. Like every other weight, the table starts out random and is learned during training.

#Measuring similarity: the dot product

We keep saying "similar vectors". To compute with that, the model needs a number for how alike two vectors are. The tool it uses everywhere is the dot product: multiply matching entries and add them up.

Here $|\mathbf{a}|$ is the length of the arrow (by Pythagoras, $\sqrt{a_1^2 + \dots + a_d^2}$) and $\theta$ is the angle between the two arrows. The second form says what the dot product means: it is large when the vectors are long and point the same way. Dividing out the lengths leaves $\cos\theta$, the cosine similarity, which ranges from $-1$ (opposite) through $0$ (perpendicular, unrelated) to $+1$ (same direction).

For example, take $\mathbf{a} = (3, 2)$ and $\mathbf{b} = (1, 3)$, the starting arrows in the figure below. The dot product is $3 \cdot 1 + 2 \cdot 3 = 9$. The lengths are $\sqrt{13} \approx 3.61$ and $\sqrt{10} \approx 3.16$, so $\cos\theta = 9 / (3.61 \cdot 3.16) \approx 0.79$: the arrows are about 38° apart and point broadly the same way.

You will meet the dot product again in the very next chapters: attention is built from dot products between tokens, and the final step of the model scores every vocabulary token by a dot product with the last vector. That last step, the unembedding, is a second table with one row per vocabulary token. Many models simply reuse the embedding table for it (weight tying), so a token is scored highly when the final vector points the same way as that token's own embedding.

#Meaning becomes geometry

Back to the claim from the start of the chapter: training gives tokens used in similar ways similar vectors. Why? Suppose "cat" and "kitten" are both often followed by "purred". The score for "purred" is a dot product with the vector the model builds from the words so far, so the easiest way to score it high after either word is to give the two words vectors that point the same way. Repeat that over billions of sentences and "cat" ends up close to "kitten", and "Paris" near "Rome". Nobody tells the model to do this; it is the cheapest way to predict the next token well.

Even more surprising, directions pick up meaning. The step from "man" to "woman" is roughly the same arrow as the step from "king" to "queen". That is the famous analogy from word2vec (Mikolov et al., 2013), an early method that learns one fixed vector per word:

Below is a hand-built 3D space with these properties. Real embedding spaces are learned and messier, but the clusters and the parallel arrows are real phenomena. (Simplified: in real word2vec spaces the analogy holds only roughly and only for some relationships, and the nearest word is searched among all words except the three you started from, as the figure also does.)

#Same word, different vectors

The embedding table gives every token exactly one vector, no matter where it appears. That is a static embedding. But "bank" in "the bank of the river" and "paid into the bank" are different words that share a spelling. The table cannot know which one you mean, so its vector for "bank" ends up as a compromise between both.

The transformer fixes this as it reads. In every layer (one transformer block; a model stacks dozens of them), attention mixes information from the surrounding tokens into each token's vector (chapter 5). By the middle layers the vector for "bank" has been pulled toward "river" in one sentence and toward "money" in the other. These contextual embeddings are what the rest of the model works with.

This is the big difference from older word vectors like word2vec, which stopped at the lookup. In a transformer, the lookup is only the starting point.

#Embeddings for search

The same idea works for whole texts. An embedding model reads a sentence or a paragraph and outputs a single vector for all of it, typically by averaging the final token vectors or by reading one special token. It is trained so that texts with similar meaning land close together, even when they share no words: "How do I reset my password?" ends up near "I forgot my login".

That is how semantic search works: embed every document once, embed the question, and return the documents with the highest cosine similarity. It is the retrieval half of retrieval-augmented generation, shown in chapter 15.

#Summary

• Token IDs are arbitrary labels, so each token is described by a vector instead: a list of numbers in which similar tokens get similar values.
• The numbers are learned, not chosen; meaning lives in the vector's direction, not in any single number.
• Each token ID selects one learned row of the embedding table, a vector with hundreds to thousands of numbers.
• The dot product and cosine similarity measure how much two vectors point the same way.
• Training arranges the space so that related tokens are close and consistent relationships become parallel directions.
• The table's vectors are static; attention turns them into contextual vectors that depend on the surrounding words.

The embedding table has one blind spot: the vector for "dog" is the same whether it is the first word or the last. The next chapter adds position.