#Backpropagation

Chapter 7 left one piece open. Gradient descent needs the slope of the loss for every parameter $\theta$, written $\partial L / \partial \theta$: how much the loss changes per small nudge of that one parameter, with all the others held still (the curly $\partial$ is what marks "the others held still"). Measuring each slope by nudging works, but it does not scale: nudge one parameter, re-run the model, compare the losses, and repeat for the next. A billion parameters means a billion forward passes, per step. Hopeless.

Backpropagation gets all of them in one backward sweep that costs only about twice as much as a single forward pass. It is nothing more than the chain rule from calculus, applied in a clever order.

#The chain rule, on a graph

Any computation, including a whole transformer, can be broken into small steps: add, multiply, apply a function. Draw each step as a node and you get a computation graph. For an LLM, the inputs on the left are the tokens and every single parameter, and the one output on the right is the loss. Here is a tiny graph with four inputs:

The intermediate results get names: $e = a \cdot b$, then $d = e + c$, then $L = d \cdot f$. Think of $a$, $b$, $c$, $f$ as parameters and $L$ as the loss. (A real cross-entropy loss is never negative; this one comes out at −8, but the mechanics are identical.) Step through the figure: first the forward pass fills in the values from $a = 2$, $b = -3$, $c = 10$, $f = -2$, then the backward pass fills in a gradient under every node.

Each node only knows its own small derivative:

($\tanh$ is an S-shaped curve that squashes any number into the range from −1 to 1.)

The chain rule says that to get the gradient of an input you multiply these local derivatives along the path back from the output:

Read it as a chain of exchange rates. Raise $a$ by 0.01 and $e = a \cdot b$ moves $b$ times as much, by −0.03. $d = e + c$ passes that on unchanged. $L = d \cdot f$ multiplies it by $f = -2$, so $L$ rises by +0.06. Rates along a chain multiply: 6 units of loss per unit of $a$. You can check it directly: $(2.01 \cdot (-3) + 10) \cdot (-2) = -7.94$, which is 0.06 above −8. That 6 is exactly the slope of chapter 7: positive, so gradient descent will lower $a$.

Backpropagation walks the graph from the output backwards and reuses every intermediate product. The gradient arriving at $e$ is computed once and then shared by both $a$ and $b$. That reuse is what makes it cheap. When a value feeds several later nodes, the gradients coming back from each are added.

#One neuron

Now a graph that is a real piece of a network. A neuron multiplies its input $x$ by a weight $w$, adds a bias $b$ (a second parameter, a plain offset), and squashes the result through tanh: $y = \tanh(z)$ with $z = w x + b$. This one has a single job, to output $t = 0.5$ for the input $x = 1.5$. Its loss is the squared error $L = (y - t)^2$, a simpler cousin of cross-entropy for a numeric output: 0 when the output is right, growing as it misses.

That makes a model with exactly two parameters, like the landscape in Figure 7.2. The curve in the figure is that landscape cut along $w$, with $b$ held still. The chain rule runs back through the three steps of the neuron:

The three factors have clear meanings:

• $2(y - t)$: how wrong the output is, and in which direction.
• $1 - y^2$: how sensitive the neuron is right now (tanh's slope).
• $x$: how much this weight actually contributed. A weight attached to a zero input gets no blame.

The middle factor is a warning. A deep network multiplies one such factor per layer on the way back, and if many of them are small, the product shrinks toward zero and the early layers stop learning. The residual connections of a transformer give the gradient a path around those factors.

#Training a real (tiny) network

Stack neurons into layers and the same rules apply to every weight. Every training step, for this toy and for an LLM alike, is the same four moves:

1. Forward: run the model on a batch and keep every intermediate value.
2. Loss: compare the predictions with the right answers to get one number.
3. Backward: walk the graph in reverse to get the slope of every parameter.
4. Update: move every parameter a small step against its slope, $\theta \leftarrow \theta - \eta \, \nabla L$, as in chapter 7.

Then repeat with the next batch. The network below has two inputs (the coordinates of a point), two hidden layers (layers between the input and the output) of 16 tanh neurons, and one output: the probability that a point belongs to the accent-colored class rather than the amber one. It starts with random weights. Press Train: each step runs forward over all points, computes the cross-entropy loss from chapter 7, backpropagates, and nudges all 337 parameters.

No one told the network what a moon or a spiral looks like. Its hidden neurons each learn a simple tilted boundary, and the next layer combines them into curves. A transformer does the same thing with far more parameters, learning features of language instead of features of a 2D plane.

#What PyTorch actually does

The engine behind Figure 8.1 is about a hundred lines of TypeScript, in the spirit of Andrej Karpathy's micrograd:

mul(other: Value): Value {
  const out = new Value(this.data * other.data, [this, other], '×');
  out.backwardStep = () => {
    this.grad += other.data * out.grad; // local derivative × incoming gradient
    other.grad += this.data * out.grad;
  };
  return out;
}

backward() {
  const order = topoSort(this);         // every node, inputs before outputs
  this.grad = 1;
  for (const v of order.reverse()) v.backwardStep();
}

PyTorch (the most widely used library for building and training neural networks), JAX and every other deep-learning framework do exactly this, with two differences that matter for speed:

• Tensors, not scalars. Each node is a whole matrix operation, so one "multiply" is a matrix multiplication over millions of numbers, run on a GPU. The local rules become matrix rules, for example the gradient of $Y = XW$ with respect to $W$ is $X^\top \frac{\partial L}{\partial Y}$.
• Memory for activations. The backward pass needs the forward values (notice that the graph above kept them). For a large model these stored activations take more memory than the weights themselves, which is why tricks like activation checkpointing recompute some of them instead of storing them.

When you write loss.backward() in PyTorch, it builds this graph as your code runs and then walks it in reverse, accumulating .grad for every parameter, just like the stepper above.

#Summary

• Backpropagation computes the gradient of the loss with respect to every parameter in one backward pass.
• It is the chain rule: multiply local derivatives along paths from the output, add where paths merge.
• Each training step is forward, loss, backward, update. Backpropagation is the backward part; gradient descent is the update.
• The same procedure trains a 337-parameter toy network and a trillion-parameter LLM; only the size of the tensors changes.

With tokens, vectors, attention and training in place, the next part looks at how a trained model is actually used, starting with how it picks a token: sampling.