Position

"Dog bites man" and "man bites dog" contain exactly the same tokens, but only one of them is news. Their embeddings are the same three vectors, only in a different order. Attention, the heart of the transformer (chapter 5), compares every token with every other token, and that comparison looks only at the vectors, not at where they sit. Left on its own, the model would read a sentence as a bag of words: it would know which words are there, but not who bit whom. (Slightly simplified: GPT-style models leak a little order through the rule that each token sees only the words before it, as the end of this chapter shows.)

Figure 4.1 · The order problem

dogpos 0bitespos 1manpos 2

dog0.90−0.300.400.10

bites−0.200.80−0.500.60

man0.500.200.90−0.70

manpos 0bitespos 1dogpos 2

man0.500.200.90−0.70

bites−0.200.80−0.500.60

dog0.90−0.300.400.10

Same rows, just reordered. Attention compares only these numbers, so “dog” matches “man” exactly as strongly in both sentences, and nothing tells the model who bit whom.

Without position, the right sentence is just a reshuffle of the same rows, so the model cannot tell the two apart. Switch to “+ position” to add a position code to each row and the inputs become genuinely different.

The fix is to mix information about where each token is into its vector. There are two broad families: add a position vector to the embeddings at the start, or build position into the attention computation itself. Modern LLMs mostly use the second.

Sinusoidal position vectors

The simplest idea is to write the position number itself into the vector: 0, 1, 2, and so on. But the numbers in an embedding are small, around ±1, and position 3,000 would be a giant number that drowns out the meaning. Dividing by the length of the text does not help either: then 0.5 means word 5 of a 10-word text but word 500 of a 1,000-word one. What we want is a code that stays small, is different for every position, and changes smoothly, so that neighbours get similar codes.

The original transformer (Vaswani et al., 2017) builds such a code out of waves. Think of a clock with many hands. The first dimensions are a second hand that spins fast; later dimensions are minute and hour hands that turn slowly. Reading all hands together gives every position a unique code, and nearby positions get similar codes. In formulas:

PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right) \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)

Here $pos$ is the position (0, 1, 2, …), $d$ is the length of the vector, and $i$ counts pairs of dimensions. Each pair holds the sine and the cosine of the same angle, which are just the two coordinates of the tip of one clock hand. For the first pair the angle grows by 1 radian (about 57°) per position: a fast hand. For later pairs the divisor $10000^{2i/d}$ is huge and the hand barely moves.

A tiny example with $d = 4$ , so two hands: one turning by 1 per position, one by $1/100$ .

Dimension	Position 3	Position 4
fast: $\sin(pos)$	0.14	−0.76
fast: $\cos(pos)$	−0.99	−0.65
slow: $\sin(pos/100)$	0.03	0.04
slow: $\cos(pos/100)$	1.00	1.00

The fast pair changes a lot from position 3 to 4, the slow pair hardly at all, and together they never repeat. This vector is added to the token's embedding, number by number, before the first block.

Figure 4.2 · Sinusoidal positional encoding

position 0 →→ 63

12position4dimension (sin)−0.61value19.9wavelength (positions)

Top: rows are dimensions, columns are positions. Bottom curve: how similar every position's code is to position 12. It peaks at 12: no other position has the same code, and close neighbours are the most alike. Hover, tap or use the arrow keys to move.

64 positions × 32 dimensions. Accent is +1, orange is −1. Low dimensions (top) flicker quickly from one position to the next; high dimensions (bottom) change slowly. The curve below compares the selected position's code with every other position.

Many models (GPT-2, BERT) instead learn a position vector for each slot, exactly like the embedding table but indexed by position. That works well, but has a hard limit: a model trained with 1,024 slots has no vector for position 1,025.

RoPE: position as rotation

Adding a position vector gives each token its absolute position: "I am word 517". But language mostly cares about relative position: an adjective right before its noun plays the same role on page 1 and on page 100. With absolute codes the model has to learn to work out distances on its own. RoPE builds the distance in directly.

Most current open models, including Llama, Mistral, Qwen and DeepSeek, use rotary position embedding (RoPE) (Su et al., 2021). Instead of adding anything to the embeddings, RoPE rotates the query and key vectors inside attention, the two vectors attention compares to decide which token looks at which, by an angle proportional to their position.

Split a vector into pairs of numbers and treat each pair as a 2D arrow. A token at position $m$ has each pair turned by $m\theta$ :

R(m\theta) = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix}

Attention compares a query at position $m$ with a key at position $n$ through their dot product, and a dot product depends only on the two arrows' lengths and the angle between them (chapter 3). Rotating does not change lengths. A query at position 5 turns by $5\theta$ and a key at position 3 by $3\theta$ , so the angle between them changes by $2\theta$ . Move the pair to positions 105 and 103 and the angle between them still changes by exactly $2\theta$ . In symbols:

\big(R(m\theta)\,\mathbf{q}\big) \cdot \big(R(n\theta)\,\mathbf{k}\big) = \mathbf{q} \cdot R\big((n-m)\theta\big)\,\mathbf{k}

The score depends only on the relative distance $m - n$ , not on where the pair sits in the text.

Figure 4.3 · Rotary position embedding on one pair of dimensions

query position m2key position n0

q turned by: 2 × θ = 52°
k turned by: 0 × θ = 0°
distance m − n: 2
q · k before: 0.747
q · k after RoPE: 0.894

Attention score q·k against relative distance m − n

Set the positions of a query and a key. Dashed arrows are the vectors before rotation, the faint arcs show how far each one turned. Press “Shift both” to slide both one position to the right: both arrows keep turning, but the shaded angle between them, and so the attention score, never changes. The chart shows the score as a function of the distance only.

A real attention head (one of several attention units working side by side) of dimension 128 has 64 such pairs, each with its own speed $\theta_i = 10000^{-2i/d}$ , the same clock-hands idea as the sinusoids. Fast pairs distinguish neighbouring tokens; slow pairs track distances of thousands of tokens.

Other ways: ALiBi and no position at all

ALiBi (Press et al., 2021) adds no vectors and rotates nothing. It subtracts a penalty proportional to the distance from every attention score: $\text{score}_{ij} - m\,(i - j)$ , where $i - j$ is how many tokens back key $j$ sits from query $i$ . Here $m$ is a fixed slope, not a position, and each head gets its own, so some heads focus on the last few tokens and others still look far back.

Figure 4.4 · ALiBi: a distance penalty on attention

Bias added to each score: −m × distance

Thecatsatonthemat

The0.00

cat−0.500.00

sat−1.00−0.500.00

on−1.50−1.00−0.500.00

the−2.00−1.50−1.00−0.500.00

mat−2.50−2.00−1.50−1.00−0.500.00

Attention from “mat” if all raw scores were equal

The3.4%

cat5.6%

sat9.2%

on15.2%

the25.1%

mat41.4%

Each head gets its own slope. Steep heads look only at the last few tokens; gentle heads still reach far back.

The matrix is the penalty added to each query–key score (rows are queries, hatched cells are the masked future that a token may not look at). The bars show how the last token would spread its attention if every raw score were equal: the steeper the slope, the stronger the pull toward recent tokens.

A surprising finding is that decoder-only models, the GPT-style kind this guide is about, can work with no position encoding at all (NoPE). The causal mask, the rule that each token sees only earlier tokens, itself leaks order: token 1 can attend to one token, token 100 to a hundred, and the model learns to read position from that. Some recent models mix NoPE layers with RoPE layers for better behaviour on long inputs.

The common thread: for language, what matters most is how far back a word was, and that is exactly what RoPE and ALiBi encode.

Reading beyond the training length

RoPE angles for distances the model never saw in training are out of its experience, so a model trained on 8k tokens falls apart at 32k. Slowing the rotations down (position interpolation, NTK scaling, YaRN) maps long distances back onto familiar angles. Chapter 12 shows how that works and what else breaks with long inputs.

Summary

Attention by itself ignores order, so position has to be injected.
Sinusoidal and learned encodings add a position vector to each embedding. Sinusoids work like clock hands at many speeds: small numbers, a unique code per position, similar codes for neighbours.
RoPE rotates queries and keys by position, which makes attention scores depend only on relative distance. It is the default in today's open models.
ALiBi penalizes distance directly, and NoPE relies on the causal mask alone.

With order taken care of, either in the vectors themselves or inside attention, we can finally look at how tokens exchange information: attention.

#Position

#Sinusoidal position vectors

#RoPE: position as rotation

#Other ways: ALiBi and no position at all

#Summary

Position

Sinusoidal position vectors

RoPE: position as rotation

Other ways: ALiBi and no position at all

Summary