The transformer block

Attention moves information between tokens. That is half the job. But attention can only blend: what a token receives is a weighted average of other tokens' values. To combine what it gathered into something new ("the word before was Eiffel and this one is Tower, so this is the landmark in Paris"), each token needs some computing of its own.

That happens inside each token, in a small neural network called the MLP (multi-layer perceptron: multiply by a matrix, bend, multiply by another matrix). Attention plus MLP, with some plumbing around them, make one transformer block, also called a layer. A model is that block repeated, with fresh weights each time, dozens of times.

One block, step by step

The block wraps both parts the same way. Take a copy of the token's vector, normalize it (rescale its numbers to a standard size, explained below), run the part on the copy, and add the result back onto the original vector. Normalizing before each part is called the pre-norm layout; almost every modern model uses it.

Press play and follow one token's vector. Notice that the strip labelled “residual stream”, the vector itself, changes only at the two ⊕ points:

Figure 6.1 · Inside a transformer block

residual stream0.71.0−0.70.6−1.41.4−0.70.5

branch

1/8

A token’s vector enters the block. The vertical line is the residual stream: the vector itself, carried from block to block.

The thick vertical line is the residual stream. Each side branch normalizes a copy of the vector, transforms it, and adds the result back at ⊕. The strips show the 8 numbers of a toy vector as it travels (the MLP numbers are computed for real; the attention update is illustrative, since one token alone has no one to attend to).

Written out, the whole block is two lines:

x \leftarrow x + \text{Attention}(\text{Norm}(x))

x \leftarrow x + \text{MLP}(\text{Norm}(x))

The residual stream

The most important design choice is the one that looks least interesting: the +. A block never replaces the vector; it reads a normalized copy and adds an update. Think of the vector as a shared notebook that every block can read and write into.

This has two big consequences:

Training works at depth. During backpropagation (chapter 8) the gradient (the correction signal for every weight) flows straight down the stream through every +, so even the first block of a 100-block model gets a clear learning signal. Without residual connections, deep networks barely train.
Blocks can specialize. Since each block only adds a little, early blocks can handle spelling and grammar while later blocks build more abstract features, and all of them write into the same space.

Normalization and activation

Two small parts keep the numbers well behaved. Normalization rescales a vector so its numbers are about 1 in size on average. The stream itself drifts as a hundred blocks add their updates to it; normalizing the copy that each part reads means every attention and MLP gets input of a familiar size, whatever happened upstream. Activation functions inside the MLP add the bend that makes the network more than one big matrix multiplication. The simplest, ReLU, is just $\max(0, x)$ ; GELU and SiLU are smooth versions that let positive numbers through and squash negative ones toward 0.

Figure 6.2 · Normalization and activation functions

input · mean 0.44 · rms 1.02

output · rms 1.00

RMSNorm only removes the overall size: drag “scale” and the output does not move, but “shift” still changes it. Real layers then multiply by a learned gain per dimension.

Normalization: drag scale and shift and compare RMSNorm with LayerNorm. Activations: all three curves are drawn, the selected one is highlighted; slide x to read off a value.

GPT-2 used LayerNorm: subtract the mean, divide by the standard deviation (the typical distance of the numbers from their mean). Most current models (Llama, Mistral, Qwen, Gemma) use the simpler RMSNorm, which skips the mean:

\text{RMSNorm}(x) = \frac{x}{\sqrt{\tfrac{1}{d}\sum_i x_i^2 + \epsilon}} \odot g

where $g$ is a learned gain per dimension, $\epsilon$ is a tiny number that prevents dividing by zero, and $\odot$ multiplies element by element. It works just as well and is cheaper.

The MLP

The MLP (also called the feed-forward network) is applied to each token on its own: no token looks at another here. The classic version expands the vector to 4× its width, applies an activation, and projects back, multiplying by a second matrix to return to the original width. The wide middle gives it room: each of its numbers can respond to a different pattern in the vector. The activation in between matters, because two matrix multiplications in a row with nothing between them collapse into one.

\text{MLP}(x) = W_2\, \text{GELU}(W_1 x)

Current models use a gated variant, SwiGLU, with three matrices: one branch is passed through SiLU and acts as a gate that multiplies the other branch element by element:

\text{SwiGLU}(x) = W_2\big(\text{SiLU}(W_1 x) \odot W_3 x\big)

In large models about two thirds of all parameters sit in these MLPs.

Stacking blocks

The block's output has exactly the shape of its input, so blocks stack. Each token keeps its own column of vectors running up through the model, and at every block attention passes information across columns.

Figure 6.3 · A stack of blocks

Columns are tokens, rows are blocks. Pulses climb the residual stream of each token; at each block, attention links (lines) carry information from earlier tokens to later ones. Information only flows rightward, toward later tokens.

At the top, a final normalization is applied, and each token's vector is multiplied by the unembedding matrix (vocabulary × $d_\text{model}$ ) to get one logit (a score) per token in the vocabulary. When generating, only the last position matters: its logits predict the next token. In training, every position predicts its own next token at once. GPT-2 reuses the embedding table for this (weight tying); most larger models keep a separate output matrix.

So here is the whole model, from text to prediction:

Split the text into tokens (chapter 2).
Look up each token's vector in the embedding table (chapter 3), and add a position vector if the model uses one (RoPE models rotate queries and keys inside attention instead, chapter 4).
Run all the vectors up through the blocks. In each block, attention lets tokens exchange information (chapter 5), then the MLP works on each token alone; both add their result to the stream.
Normalize once more and multiply by the unembedding matrix: one logit per vocabulary token.
Softmax turns the last position's logits into probabilities, and one token is picked (chapter 9). Append it and start again.

Watching the answer form

Since every block only adds to the residual stream, the stream always lives in the same space. So we can cheat: take the stream after block 3, or block 7, apply the final norm and unembedding early, and ask "what would the model say now?" This trick is called the logit lens.

Figure 6.4 · The logit lens

The·Eiffel·Tower·is·in·the·city·of·Paris?

after block 4

·New7.0%

·the6.0%

·London5.0%

·Rome2.0%

·Paris1.5%

Probability of “Paris” after each block

Middle blocks: the model has figured out a city is coming. Cities compete.

The next-token guess after each block of a 12-block model, for a prompt that ends “…the city of”. Press the button or drag the slider. Early blocks guess grammatical filler, middle blocks narrow down to cities, upper blocks commit to the answer. The numbers are hand-made, but this shape is what the logit lens typically shows on real models.

The picture fits what interpretability research finds: lower layers handle local, surface features (spelling, grammar, the previous few tokens), middle layers assemble meaning and look up facts, and upper layers turn that into a confident prediction.

What the MLP stores

Where does a fact like "the Eiffel Tower is in Paris" live? Experiments that edit or knock out individual MLP layers point to the middle MLPs. A useful picture is a key-value memory: the rows of $W_1$ are keys that match patterns in the stream ("this is about the Eiffel Tower"), and the matching columns of $W_2$ are values that write the associated information ("Paris, France") back into it.

It is rarely that clean, because models store far more features (concepts they can detect) than they have dimensions. A 4096-dimensional stream can hold many thousands of nearly perpendicular directions, and models exploit this. This is called superposition:

Figure 6.5 · More features than dimensions

cat0.00

Paris0.65

code0.00

past tense0.00

sarcasm0.00

Clean readout: only the active features light up.

Closest pair: 72° apart · ● hidden vector h

Each arrow is a feature stored as a direction in a space of only 2 numbers. Switch features on and watch the readouts: with 3 to 5 features and one active at a time, a small threshold (ReLU with a bias) reads them back cleanly. Activate two non-neighbors in the pentagon, or squeeze in 6 or more, and features that are off start to light up while active ones can cancel out. Superposition works because most features are rarely active at the same time.

This is also why single neurons, the individual numbers in the MLP's wide middle layer, are often hard to interpret: one neuron takes part in many features, and one feature is spread over many neurons. Interpretability researchers train sparse autoencoders (small networks that rewrite the stream as a handful of active features at a time) to pull the features back apart, and have found millions of them in production models, from "Golden Gate Bridge" to "code with a security bug". Turning such a feature up or down measurably changes the model's behavior.

How big is it?

Every number in this chapter is a parameter that training has to learn. Counting them is simple arithmetic, so the calculator below gives exact totals.

Figure 6.6 · Counting parameters

124.4Mparameters (124,439,808)0.2 GBto store at 16 bits each3,072MLP hidden width

Embeddings 31.6% 39.4M
Attention 22.8% 28.3M
MLP 45.5% 56.7M
Norms 0.0% 38.4K

Per layer: attention 4d² + 4d, MLP 8d² + 5d, two LayerNorms 4d. Plus token and position tables (the token table is reused as the output layer).

Pick a preset, then change the depth, width and vocabulary. GPT-2 small comes out at exactly 124,439,808 parameters and Llama 2 7B at 6,738,415,616. Notice how the MLP dominates as models get wider, and how much of a small model is just its embedding table.

Two rules of thumb fall out of the formulas. Each layer holds about $12\, d_\text{model}^2$ parameters, so doubling the width quadruples the cost of a layer. And at 16 bits per parameter, a model needs about 2 bytes of memory per parameter just to be loaded: a 7B model needs about 14 GB before processing a single token. Newer models shrink parts of this, for example with grouped-query attention (chapter 10) and quantization (chapter 13).

Summary

A block is two residual updates: attention (between tokens), then an MLP (within each token), each applied to a normalized copy.
The residual stream carries each token's vector up through the model; blocks only add to it, which keeps deep models trainable.
RMSNorm keeps vectors at a stable size; GELU and SwiGLU give the MLP its non-linearity.
The whole model: tokens → embeddings → a stack of blocks → final norm → unembedding → softmax over the vocabulary.
The final vector is turned into logits by the unembedding matrix; the logit lens applies it early to watch the answer form layer by layer.
MLPs act partly as key-value memories, and models pack more features than dimensions through superposition.

That is the entire architecture. Everything that makes it smart comes from the values of its parameters, and those come from training.

#The transformer block

#One block, step by step

#The residual stream

#Normalization and activation

#The MLP

#Stacking blocks

#Watching the answer form

#What the MLP stores

#How big is it?

#Summary