What an LLM does

A large language model has exactly one skill: given some text, it predicts what comes next. Not a whole answer, not a sentence. One small piece of text, called a token.

Everything else, from chatting to writing code to "reasoning", comes from running that one prediction in a loop.

Figure 1.1 · The generation loop

The·cat·sat·on·the

Read
Layers
Scores
Pick
Append

·mat0.0%

·floor0.0%

·sofa0.0%

·roof0.0%

·dog0.0%

…and about 200,000 other tokens sharing the remaining 13%.

1/25

The model sees 5 tokens. It never sees letters, only these pieces (as numbers).

Press play. Each round the model reads everything so far, scores every token in its vocabulary (the fixed list of tokens it knows), picks one, and appends it. Then it starts again with the slightly longer text.

The loop, in five steps

Tokenize. The text is cut into tokens, pieces that are often whole words (·cat, where · marks the space in front of the word), sometimes word fragments (izers), and each piece becomes a number. Chapter 2 shows how.
Embed. Each number is swapped for a long list of numbers, a vector, that the model has learned to associate with that token. Chapter 3.
Transform. The vectors pass through dozens of transformer blocks, all built the same way but each with its own learned numbers. Inside, attention lets every token gather information from the tokens before it. Chapters 5 and 6.
Score. The last vector is compared with a learned vector for every token in the vocabulary (with a dot product) to produce a score (a logit) for each. Softmax turns scores into probabilities that add up to 1 (how, just below).
Sample. One token is drawn from that distribution, appended to the text, and the loop runs again. Chapter 9 shows how temperature changes the draw.

From scores to probabilities: softmax

The model's raw output is one score per token in the vocabulary. These scores, called logits, can be any numbers: positive, negative, large or small. To pick a token we need probabilities instead: numbers between 0 and 1 that add up to 1. Softmax is the standard way to turn the first into the second, and it comes back many times in this guide.

It takes two steps:

Exponentiate every score, computing $e^{z}$ (with $e \approx 2.718$ ). Every result is positive, and differences are stretched: a score just 1 higher ends up about 2.7 times bigger.
Normalize: divide each result by the sum of all of them, so together they add up to 1.

\text{softmax}(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

For example, take three scores, 2, 1 and 0. Exponentiating gives $e^2 \approx 7.39$ , $e^1 \approx 2.72$ and $e^0 = 1$ , which sum to about 11.11. Dividing by that sum gives probabilities of about 0.67, 0.24 and 0.09.

Figure 1.2 · Softmax, step by step

tokenscore z (logit)eᶻprobability

·mat3.020.156.8%

·sofa2.18.1723.1%

·floor1.75.4715.5%

·chair0.51.654.7%

sum of eᶻ = 35.37each probability = eᶻ ÷ this sum

Drag a score and watch its eᶻ and every probability change. '+1 to every score' leaves the probabilities exactly as they were; 'Double the gaps' makes the leader take far more.

Four things to remember about softmax:

Order is kept. A higher score always gets a higher probability, and no token ever gets exactly zero.
Only differences matter. Adding the same number to every score changes nothing, because the common factor cancels out in the division.
Gaps are amplified. Larger gaps between scores push almost all of the probability onto the top token. That is why attention divides its scores by $\sqrt{d}$ (chapter 5), and what temperature controls when sampling (chapter 9).
It is a "soft" max. A hard max would give the top score 100% and everything else 0%. Softmax gives the top score the biggest share and keeps a little for the rest, which also lets training nudge every score a bit (chapter 7).

Written as math

If the text so far is the tokens $x_1, \dots, x_t$ , the model computes a probability for every possible next token:

P(x_{t+1} \mid x_1, \dots, x_t)

Read it as "the probability that the next token is $x_{t+1}$ , given ( $\mid$ ) all the tokens so far". The probability of a whole sentence is these next-token probabilities multiplied together:

\begin{aligned} P(\text{The cat sat}) = {} & P(\text{The}) \\ {} \cdot {} & P(\text{cat} \mid \text{The}) \\ {} \cdot {} & P(\text{sat} \mid \text{The cat}) \end{aligned}

Generation walks the same chain forward: draw one token, append it, and condition the next draw on it. A model that feeds each output back in as part of the next input is called autoregressive.

The context window

The "text so far" has a hard limit: the context window, the maximum number of tokens the model can look at in one pass. In English a token is about four characters on average, so 1,000 tokens is roughly 750 words; most other languages need more tokens for the same text (chapter 2). Everything the model knows about your conversation has to fit inside it: the system instructions, every earlier message, pasted documents, and its own replies. Anything that falls outside simply does not exist for the model. It has no other memory of the chat.

Figure 1.3 · What does not fit is forgotten

Context window52 tokens

systemYouareahelpfulassistant.

userMynameisAda.IliveinLisbon.

assistantNicetomeetyou,Ada!HowcanIhelp?

userRecommendacafénearme.

assistantTryapastelariainAlfama,theymakegreatpastéisdenata.

userWhatismyname?

The name "Ada" is inside the window: the model can answer.

Shrink the window. The oldest tokens drop out first, and once every mention of the name is gone, the model has no way to answer the last question. (Here one word counts as one token; real tokens are a bit smaller.)

Windows have grown fast: GPT-2 (2019) saw 1,024 tokens, GPT-3 2,048, the first GPT-4 8,192 or 32,768, and today's models handle 128,000 to over a million. Bigger windows are not free, though: the cost of attention grows with the square of the length, and models often use the middle of a long context worse than its beginning and end. Chapter 12 is about exactly that, and chapter 11 about not paying for the same long prompt (the text sent to the model) twice.

Where the knowledge lives

The model's knowledge sits in its parameters: billions of numbers arranged in weight matrices (rectangular grids of numbers). Small open models, whose weights anyone can download, have one to a few billion; frontier models, the largest and most capable of the moment, have hundreds of billions or more. Nobody writes these numbers by hand. They start random and are nudged, trillions of times, toward values that make the next-token guesses better. Chapters 7 and 8 show that nudging, called training.

The surprising part

Nothing in this loop says "answer questions" or "be helpful". A model trained only to continue internet text becomes a capable base model, and a smaller amount of extra training turns it into an assistant. Chapter 15 covers that last step.

The map of this guide

Part	Chapters	What you will see
From text to numbers	1–4	Tokens, vectors, and how position is encoded
Inside the transformer	5–6	Attention and the block that repeats
Learning	7–8	Loss, gradient descent, scaling laws, backpropagation, live training
Using the model	9–12	Sampling, fast inference (running a trained model), caching, long context
Modern techniques	13–15	Mixture of experts, quantization, LoRA, new architectures, RLHF, DPO, reasoning, RAG

Every figure is live: press play, drag the sliders, type your own text. The examples are deliberately tiny, a handful of tokens and a few dimensions, but the math is the same math a frontier model runs.

We start where the loop starts, with step 1: how text is cut into tokens, in chapter 2.

#What an LLM does

#The loop, in five steps

#From scores to probabilities: softmax

#Written as math

#The context window

#Where the knowledge lives

#The map of this guide