#Long context

The context window is the number of tokens a model can take into account at once: the prompt, any documents, the conversation so far and its own output. It grew from 2,048 tokens in GPT-3 to 128,000 in many open models today, and some commercial models accept a million or more. Long context lets a model read a whole codebase or a book in one go. It also brings three problems: cost, accuracy, and the position encoding itself.

#Problem 1: cost

Two things grow with the context length $n$:

• Memory. The KV cache stores keys and values for every token: it grows linearly, $\propto n$.
• Compute. Every token's query is compared with every earlier key. The number of pairs grows quadratically, $\propto n^2$.

All the other matrix multiplications (projections and MLPs) cost the same per token, so they grow linearly. Prefill compute is roughly

for $P$ parameters, $L$ layers and model width $d$ (the length of each token's vector). The first term is one multiply and one add per parameter per token. The second counts attention itself: about $n^2/2$ pairs of tokens (each token looks only at earlier ones), and in every layer each pair costs a dot product of length $d$ for its score and another $d$ multiply-adds to mix in its value, about $4d$ operations.

For short prompts the first term wins. Past the length where the two are equal, $n = P / (L d)$, attention dominates and the cost curve bends upward. For Llama 3 8B that is $8 \times 10^9 / (32 \times 4096) \approx 61{,}000$ tokens.

Decode feels the length too, through memory rather than arithmetic. Every decode step reads the whole cache (chapter 10 explains why reading, not computing, sets decode speed). At 128k tokens the cache of an 8B model is as big as its weights, so each new token takes roughly twice as long as with a short prompt.

Grouped-query attention, KV quantization and the other tricks from Caching attack the memory line. FlashAttention (Fast inference) makes the quadratic compute fast and memory-light, but still quadratic. Changing the exponent itself needs a different attention pattern; more on that below.

#Problem 2: does the model actually use it?

A model that accepts a million tokens does not necessarily use them well. Two experiments made this visible.

Lost in the middle (Liu et al., 2023) gave models 20 retrieved documents, only one of which held the answer, and moved that document around. Accuracy was highest when it came first or last and dropped sharply in the middle, at worst close to answering with no documents at all. Needle in a haystack hides one out-of-place sentence at a chosen depth in a long filler text and asks for it back, sweeping both depth and length.

Newer models pass the simple needle test almost perfectly, so harder benchmarks (RULER, multi-needle tests, and multi-hop tests whose questions need several facts chained together) are used instead. They still find that the effective context, the length at which a model keeps its short-context quality, is often well below the advertised one, and that quality slowly degrades as more irrelevant text is added. People call this context rot.

Why the U-shape? The causes are still debated; three are usually given. Training data rarely needs information from the middle of very long texts; attention patterns favor the start (the "attention sink" we saw in Caching) and the most recent tokens; and with causal attention, early tokens are processed without knowing what the question will be.

#Problem 3: positions the model has never seen

RoPE encodes position by rotating pairs of numbers in each query and key, every pair at its own speed. Fast pairs complete many full turns within the training length; slow pairs complete less than one. A model trained on 4,096 tokens has never seen the slow pairs turned any further than their 4k angle. Feed it position 30,000 and those pairs land on angles it has never seen, and attention breaks.

The fixes all squeeze positions back into the trained range:

• Position interpolation (PI) divides every position by $s =$ target length / training length. All angles fit, but neighbours are now $s$ times closer, which blurs the fast pairs.
• NTK-aware scaling (named after a piece of theory about how networks learn, the neural tangent kernel) increases the RoPE base instead (the 10,000 that sets the rotation speeds, chapter 4), so fast pairs barely change and slow pairs stretch the most.
• YaRN treats each pair separately: pairs that already turned many times are left alone, slow ones are interpolated, with a ramp in between, plus a small correction to attention temperature, a factor that makes attention's softmax a little sharper, much like sampling temperature. Many open long-context models use YaRN or a close relative.

Scaling alone is not enough for top quality. In practice models are pretrained at a moderate length (4k to 8k), then continue training on long documents while the scaled context grows in stages, up to 128k or beyond. That last phase is a small fraction of the training compute but teaches the model to actually use distant tokens.

#Cheaper attention patterns

To change the quadratic exponent, a token must stop looking at every earlier token. Sparse attention keeps a structured subset. Chapter 5 showed which scores these patterns skip; the question here is reach. If a token cannot see far back directly, can information from far back still get to it?

• Sliding window attention (used by Mistral 7B, and interleaved with full-attention layers in models like Gemma) costs $O(n \cdot w)$, work that grows with the length times the window rather than the length squared, and keeps the KV cache at $w$ tokens per layer.
• Global or sink tokens add a few positions everyone can see (Longformer, BigBird, StreamingLLM).
• Learned sparse attention lets the model pick which blocks of earlier tokens to read for each query, as in DeepSeek's sparse attention and Native Sparse Attention.
• Some architectures drop attention's growing cache altogether and keep a fixed-size state instead: see Beyond the classic transformer.

Very long contexts also need more than one GPU per sequence. Ring attention and context parallelism split the sequence across devices; each device holds a slice of the queries while blocks of keys and values are passed around the ring, overlapping the transfer with computation.

#Long context in practice

• Long context or retrieval? Stuffing everything into the prompt is simple and lets the model connect distant facts, but it is slow and expensive and suffers from context rot. Retrieval (RAG) sends only the relevant pieces. Many systems combine both: retrieve generously, then let a long-context model read.
• Put the question last, and important material near the start or end. Lost-in-the-middle effects are smaller in new models, but rarely zero.
• Cache the long, stable part. A big document followed by many questions is the ideal case for prompt caching.
• Agents compact their history. Long-running agents summarize or drop old tool outputs, keep notes in files, and start fresh contexts for sub-tasks, instead of letting the context fill with noise.

#Summary

• The KV cache grows linearly with context, attention compute quadratically; beyond some tens of thousands of tokens attention dominates.
• Accepting a long context is not the same as using it: recall dips in the middle and degrades with length.
• RoPE needs scaling (PI, NTK-aware, YaRN) plus continued training on long documents to work past its training length.
• Sparse patterns, sliding windows and state-based architectures attack the quadratic cost; ring attention spreads one sequence over many GPUs.

Long context is one way a model gets expensive; sheer size is the other. The next chapter shows how to make big models cheaper to run and to adapt: mixture of experts, quantization and LoRA.