#Loss, gradient descent, scaling

A freshly created model is a pile of random numbers. Ask it to continue "The cat sat on the" and every token in its ~200,000-token vocabulary gets roughly the same tiny probability. Training is the process of changing those numbers until the predictions become good. It needs two things:

1. A single number that says how bad the predictions are right now: the loss.
2. A way to change billions of parameters so that number goes down: gradient descent.

#What training changes

Those random numbers are the parameters, also called weights, that filled the last four chapters: every entry of the embedding table, of the query, key and value matrices in every attention head, of the two MLP matrices in every block, and of the unembedding matrix at the top. Each parameter is one plain number, such as 0.013 or −0.41. A 7B model has about seven billion of them; Figure 6.6 counts where they sit.

Running the model is a long but fixed calculation: token IDs go in, one probability per vocabulary token comes out, and the parameters are the constants inside that calculation. Nothing else in the model can change, so "learning" can only mean changing the parameters.

Every parameter moves the output a little. Take the very last step: the logit of "·mat" is the dot product of the final vector with the row of the unembedding matrix that belongs to "·mat". Nudge one number in that row and the logit shifts, softmax redistributes, and the probability of "·mat" goes up or down a hair. A weight deep inside block 3 has the same kind of effect, only through a much longer chain of arithmetic. Training has to find out, for each of the billions of parameters, which way to nudge it. First it needs an exact definition of "better".

#Measuring "wrong": cross-entropy

Take a sentence from the training text, hide the next token, and ask the model for its probability distribution. The loss only looks at one number in that distribution: the probability $p$ the model gave to the token that actually came next.

$$\text{loss} = -\ln p(\text{correct token})$$

That is the cross-entropy loss. If the model gave "·mat" a probability of 0.2, the loss is $-\ln 0.2 \approx 1.61$. Raise that probability to 0.9 and the loss drops to 0.11; lower it to 0.001 and the loss jumps to 6.9. The figure starts at 0.2: drag the slider to see why the minus-log shape is a good choice.

Figure 7.1 · Cross-entropy for one prediction

The cat sat on the ___ correct next token: ·mat

·mat20.0%

·floor32.0%

·sofa24.0%

·roof16.0%

·dog8.0%

P(·mat)0.20

1.61loss, −ln P(·mat)5.0perplexity, 1 / p

Mostly wrong: the penalty climbs fast.

A few things worth noticing:

• Only the correct token's probability enters the formula, but because all probabilities must add to 1, raising it automatically lowers the wrong ones.
• A model that guesses uniformly over a 200,000-token vocabulary scores $\ln 200{,}000 \approx 12.2$. That is roughly the loss you see at step zero of a real training run.
• Perplexity is $e^{\text{loss}}$: "the model is as unsure as if it were choosing among this many tokens". A loss of 2 means a perplexity of about 7.4.

In practice the loss is averaged over every position of every sequence in a batch, often millions of tokens at once. One text of 1,000 tokens already gives 1,000 separate next-token predictions to learn from.

That average is the loss: one number that scores the whole model on this batch. Keep the batch fixed and the loss depends only on the parameters. Change any of them and the probabilities change, and with them the loss. Training is a search for parameter values that make it small.

#Rolling downhill

Start with a single parameter and hold all the others still. Nudge it up by a tiny amount, say 0.001, and measure the loss again. Suppose the loss rose by 0.002: it climbs at a rate of 2 per unit of this parameter. That rate is the parameter's slope, here $+2$, and it says "to lower the loss, move this parameter down". A slope of $-0.5$ would say "move it up; it matters a quarter as much". A slope near 0 says this parameter hardly matters right now.

Collect one slope per parameter and you have the gradient, $\nabla L$: a list of billions of numbers which, read as one direction, points the way the loss rises fastest. So we step the other way, moving every parameter against its own slope, in proportion to it:

$$\theta \leftarrow \theta - \eta \, \nabla L(\theta)$$

Here $\theta$ stands for all the parameters and $\eta$ (eta) is the learning rate, the size of each step. With a slope of $+2$ and $\eta = 0.01$, that parameter moves down by 0.02. Measuring slopes by nudging one parameter at a time would cost a full run of the model per parameter, billions of runs per step; backpropagation, the next chapter, gets all of them at once. For now, assume we have them.

The usual picture is a landscape. Every possible setting of the parameters is a location, and the loss is the height of the ground there. Training means walking to a low point, but the model cannot see the landscape, only the slope under its feet, so it walks downhill one step at a time. A real model's landscape has one direction per parameter and cannot be drawn. A model with only two parameters can: below, the two parameters are the two horizontal directions of the surface, and the height is the loss. Three optimizers, rules for turning a gradient into a step (described just below), start from the same spot.

Figure 7.2 · Gradient descent on a loss surface

#The three optimizers

• SGD (stochastic gradient descent) takes the step exactly as written above. "Stochastic" because each step uses the gradient of a random batch, not of the entire dataset, so the direction is a noisy estimate.
• Momentum keeps a running sum of past gradients, like a heavy ball that keeps rolling. It speeds up along consistent directions and damps the zig-zag in narrow valleys.
• Adam adds a second running average, of the squared gradient, and divides each parameter's step by its square root. Parameters with large, noisy gradients take careful steps; quiet ones take bigger steps. Adam, and its variant AdamW with weight decay (a small pull of every weight toward zero at each step), trains almost every modern LLM.

#Why a real landscape is friendlier than it looks

Real loss surfaces have billions of dimensions, not two. That sounds hopeless, but it helps: a true local minimum needs the ground to curve upward in every direction at once, which is rare. Most flat-looking spots are saddle points with some downhill direction left, and the noise from random batches keeps the optimizer moving. Large models trained with the right settings reach very similar loss values from different random starts.

#A real training run

Two more pieces turn the textbook update into how LLMs are actually trained.

The learning-rate schedule. At the start, the weights are random and the gradients are large and erratic, so a full-size step can throw the model somewhere awful. Training therefore begins with a warmup: the learning rate rises linearly from zero over the first few thousand steps. Afterwards it slowly decays, typically along a cosine curve, so the final steps are small and precise.

The batch size. The gradient from a batch is an average over its examples. Bigger batches give a cleaner estimate (the noise shrinks like $1/\sqrt{\text{batch}}$) and let thousands of GPUs work in parallel, at the price of fewer steps per token seen.

Figure 7.3 · Learning-rate schedule and loss curve

Warmup steps40Batch size32

Final loss ≈ 2.62. Small batches give noisy, jagged curves; skipping warmup makes the first steps blow the loss up before it recovers.

The shape of the loss curve, steep at first and then a long, slow tail, is extremely consistent from run to run. That consistency is what makes the next section possible.

#Scaling laws

If you plot the final loss of many training runs against the compute they used (the total number of arithmetic operations), a remarkably clean pattern appears: loss falls as a smooth power law, where every tenfold increase in compute shrinks the loss by about the same factor. (Strictly, it is the part of the loss above an unbeatable floor that shrinks this way; the formula below makes that floor explicit.) On log-log axes (both marked 1, 10, 100, …) that is a straight line, across many orders of magnitude. Labs fit these curves on small, cheap runs and use them to predict how a model a hundred times bigger will do before spending the money.

Training compute has a simple formula, counted in FLOPs (floating-point operations: single additions or multiplications). For a model with $N$ parameters trained on $D$ tokens:

$$C \approx 6\,N\,D \quad \text{FLOPs}$$

(about 2 operations per parameter per token for the forward pass, running the model on its input, and 4 for the backward pass that computes the gradients). So for a fixed budget $C$ there is a trade-off: a bigger model sees fewer tokens. Which split gives the lowest loss? The Chinchilla study (Hoffmann et al., 2022) fitted the loss as

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

where $E$ is the irreducible entropy of text (the unpredictability no model can remove), and the two other terms are the penalties for a model that is too small and for data that is too scarce. Minimizing it under $C = 6ND$ says both should grow together, roughly as $\sqrt{C}$, which works out to about 20 training tokens per parameter.

Figure 7.4 · Compute-optimal training

Compute budget10²³Model size vs optimal1/8

Compute-optimal29.4B parameters
566B training tokens
≈ 19 tokens per parameter
final loss 2.032

Your choice3.68B parameters
4.53T training tokens
≈ 1230 tokens per parameter
final loss 2.093 · cost per generated token ×0.13

8.0× cheaper to run, for +0.061 loss. With billions of users, that trade is usually worth it.

Before Chinchilla, many large models were too big for the data they saw. Chinchilla itself, 70 billion parameters trained on 1.4 trillion tokens, beat a 280-billion-parameter model trained with the same compute.

Why modern models ignore the rule. "Compute-optimal" only counts the cost of training. A deployed model then generates trillions of tokens, and every one of them costs about $2N$ FLOPs. A smaller model trained on far more data is slightly worse than the optimum for its training budget, but much cheaper forever after. That is why open models are now routinely trained far past 20 tokens per parameter: Meta's Llama 3 8B saw over 15 trillion tokens, close to 2,000 per parameter. The loss keeps falling, just more slowly.

#Training at scale

A frontier training run spends something like $10^{25}$ FLOPs and more. No single GPU could do that in a lifetime, and no single GPU has the memory to hold the model. Two tricks make it possible: cheaper numbers and many GPUs.

#Mixed precision

Every weight, activation (an intermediate value computed inside the model) and gradient is a floating-point number. Fewer bits per number means less memory, less data to move, and much faster matrix units. The catch is that a float splits its bits between an exponent (how large or small a number can be) and a mantissa (how many significant digits it keeps). Type a number to see what each format stores.

Figure 7.5 · The number formats of training

Number to storeTry:π0.1700001e-8300

sign exponent (range) mantissa (precision)

FP32 · 32b3.1415927error 0.0000028%max 3.4e+38

BF16 · 16b3.140625error 0.031%max 3.4e+38

FP16 · 16b3.140625error 0.031%max 65,504

FP8 E4M3 · 8b3.25error 3.5%max 448

FP8 E5M2 · 8b3error 4.5%max 57,344

BF16 keeps FP32’s 8 exponent bits, so tiny gradients and big activations both fit; it pays with only 3 significant digits. FP16 is more precise but overflows above 65,504 and flushes tiny values to zero, which is why FP16 training needs “loss scaling”.

Modern LLMs train in mixed precision: the big matrix multiplications run in BF16, a 16-bit format with the range of standard 32-bit FP32 but fewer digits (and increasingly FP8, with a scale factor per block of numbers to keep values in range), while a master copy of the weights and the optimizer state stay in FP32 so that tiny updates are not rounded away. With Adam this costs about 16 bytes per parameter: 2 for the BF16 weights, 2 for the gradients, 4 for the FP32 master weights and 8 for Adam's two running averages. A 70-billion-parameter model therefore needs over a terabyte of memory before counting any activations.

#Spreading the work across GPUs

That terabyte, and the compute, are split across thousands of GPUs in three complementary ways:

Figure 7.6 · Data, tensor and pipeline parallelism

DataTensorPipeline

Every GPU holds a full copy of the model and gets a different slice of the batch.

• Data parallelism is the simplest: every GPU has the whole model and a different slice of the batch, and the gradients are averaged with an all-reduce (an exchange after which every GPU holds the same average) after every step. Its modern form, ZeRO or FSDP (fully sharded data parallel), also shards the weights, gradients and optimizer state, splitting them into one piece per GPU, and gathers each layer only when it is needed, so no GPU holds the full 16 bytes per parameter.
• Tensor parallelism cuts each weight matrix into column or row blocks (a tensor is a block of numbers with any number of dimensions; a matrix is a 2-D tensor). Every layer then needs an exchange of partial results, so it is used among the 4–8 GPUs inside one server, connected by very fast links.
• Pipeline parallelism gives each group of GPUs a range of layers and streams micro-batches through them like an assembly line. Its cost is the bubble: stages sit idle while the pipeline fills and drains.

Two more axes appear in recent models: expert parallelism puts different experts of a mixture-of-experts layer on different GPUs, and context (sequence) parallelism splits very long sequences across GPUs for long-context training.

#Data

Classic machine learning loops over a dataset many times; each pass is an epoch. LLM pretraining, the first and by far largest stage of training (next-token prediction on raw text), usually sees most documents about once. The budget is counted in tokens instead: small open models are trained on many trillions of tokens. At that scale the quality of the data matters as much as its quantity, and a large share of the engineering goes into it:

• Deduplication. The web is full of copies. Exact and near-duplicate documents (found by comparing short fingerprints of each text, a hashing trick called MinHash) are removed, because repeated text wastes compute and encourages memorization.
• Filtering. Spam, boilerplate, and machine-generated junk are dropped. Increasingly, small classifier models (models that sort inputs into categories) score each page for educational value, and only the best fraction is kept.
• Mixing. The final blend is a careful recipe of web pages, code, books, papers, math and many languages, with weights tuned on small proxy runs.
• Synthetic data. Existing models write textbooks, exercises, worked solutions and rephrased versions of good documents. Used carefully, this adds high-quality tokens where natural text is scarce.
• Annealing. Near the end of pretraining, as the learning rate decays to zero, the mixture shifts toward the highest-quality data (curated math, code, reasoning). These last steps have an outsized effect on the final model. Some labs call a dedicated phase like this mid-training, which may also stretch the context length.

#Summary

• The parameters are the numbers in the model's tables and weight matrices. Training changes nothing else.
• The loss for next-token prediction is cross-entropy: $-\ln$ of the probability given to the correct token, averaged over a batch.
• A parameter's slope says how the loss reacts when that parameter is nudged; the gradient is the list of all the slopes. Gradient descent moves every parameter a small step against its slope; the learning rate sets the step size. LLMs are trained with AdamW, a warmup, and a decaying schedule.
• Loss falls as a power law in compute. For a fixed budget, about 20 tokens per parameter minimizes training loss, but deployed models are trained far longer because smaller models are cheaper to run.
• Training at scale relies on BF16/FP8 mixed precision and on data, tensor and pipeline parallelism across thousands of GPUs, and on carefully deduplicated, filtered and mixed data.

One question is left: with billions of parameters, how do we get the gradient for every single one of them? That is backpropagation.