#Beyond the classic transformer

The transformer from chapter 6 has carried language models since 2017, but it has two expensive habits. Every new token reads the keys and values of every earlier token, and those keys and values pile up in a KV cache that grows with the conversation. And it writes text strictly one token per forward pass.

Most recent architecture work attacks one of those habits. This chapter walks through the main ideas: shrinking the cache, replacing attention with a fixed-size memory, mixing the two, writing many tokens at once, and feeding the model things that are not text at all.

#Multi-head latent attention: compress the cache

Chapter 10 showed two ways to shrink the KV cache: let several query heads share one key/value head (GQA), or let all of them share a single one (MQA). Sharing saves memory but throws away some of what separate heads could express.

Multi-head latent attention (MLA), introduced with DeepSeek-V2 in 2024, compresses instead of sharing. For each token the model computes one small latent vector $c$ ("latent" means hidden: a compressed summary the model uses internally) and caches only that. When attention needs keys and values, it up-projects them from the latent for every head, multiplying by a matrix that makes the vector large again:

Here $h_t$ is the token's hidden vector, $W_\text{down}$ squeezes it into the latent, and each head $i$ has its own matrices $W_K^{(i)}$ and $W_V^{(i)}$ to rebuild its key and value. The sizes show the gain. In the model of the figure below, close to DeepSeek-V3, plain multi-head attention caches 2 × 128 heads × 128 numbers = 32,768 numbers per token in every layer. MLA caches a 512-number latent plus a 64-number position key: 576 numbers, about 57 times fewer.

Rebuilding every key for every head at every step would be slow, but it is never needed. A key is only ever used in a dot product with a query, and $q \cdot (W_K c) = (W_K^\top q) \cdot c$: multiply the query by $W_K^\top$ once, and it can be compared with the cached latent directly. The value matrix folds into the output matrix the same way. Rotary positions (chapter 4) do not survive that folding, so MLA keeps a small extra key per token just for position.

With the settings above, a 128k-token conversation needs roughly 490 GB of cache with plain multi-head attention, about 31 GB with GQA, and under 9 GB with MLA. That difference decides how many users one GPU server can hold at once.

#State-space models: a memory that does not grow

Attention keeps everything: to produce token 100,000 it looks back over 99,999 cached entries. A recurrent network does the opposite. It squeezes the past into a fixed-size state and updates that state once per token. Memory and work per token stay constant, however long the text gets.

Classic RNNs (recurrent neural networks, the step-by-step design just described) lost to transformers for two reasons: they could not be trained in parallel, and they forgot too much. Modern state-space models (SSMs) such as Mamba (2023) fix both.

• Selective updates. Each step is $$h_t = \bar{A}_t \odot h_{t-1} + \bar{B}_t\, x_t, \qquad y_t = C_t\, h_t$$ Here $h_t$ is the state after token $t$ (a short list of numbers), $x_t$ is the token's input, $\odot$ multiplies two lists number by number, and $y_t$, the layer's output, is read out of the state through $C_t$. The decay $\bar{A}_t$ and the write strength $\bar{B}_t$ depend on the current token. With a one-number state and $\bar{A} = 0.9$, each step keeps 90% of what was there, so after 10 filler words $0.9^{10} \approx 0.35$ of an old memory remains. An important word can get $\bar{A} \approx 0$ and a large $\bar{B}$: forget that slot and write the new word in its place. In Mamba both come from one token-dependent number, the step size Δ: a large Δ means forget more and write more.
• Parallel training. Each step is an affine map $h \mapsto a\,h + u$ (multiply by a number, then add one). Doing $h \mapsto a_1 h + u_1$ and then $h \mapsto a_2 h + u_2$ gives $h \mapsto (a_2 a_1)\,h + (a_2 u_1 + u_2)$: again one multiply and one add. So neighbouring steps can be merged in pairs, all at once, then pairs of pairs, and so on. Because of that, the steps can be grouped in any order without changing the result (associativity), which lets a parallel scan process a whole training sequence in about $\log_2 n$ rounds instead of $n$ sequential steps.

Linear attention reaches the same place from the other side. Drop the softmax from attention, and the output for a query $q$ becomes $\sum_i (q \cdot k_i)\, v_i = q^\top \big(\sum_i k_i v_i^\top\big)$. The sum in brackets does not depend on the query, so it can be kept as one running matrix, updated once per token: $S_t = S_{t-1} + k_t v_t^\top$. Again a fixed-size state. The price is that without softmax, attention can no longer focus sharply on a few tokens, which recent variants partly win back with gates. Mamba-2, RWKV, RetNet and Gated DeltaNet are all variations on "attention-like quality with a recurrent-sized memory". Mamba-2 is in fact built on the observation that a structured SSM and a form of linear attention compute the same thing.

The catch is in the word fixed. A state of a few thousand numbers per layer cannot hold a 100-page document word for word. Pure SSMs are good at the gist and weaker at exact recall: copying a phone number from far back, or finding the needle in a long context.

#Hybrids: a few attention layers are enough

So most new long-context designs keep some attention. A hybrid stack makes most layers cheap (recurrent, linear, or sliding-window attention) and spends full attention on a few. AI21's Jamba (2024) used one attention layer for every seven Mamba layers; several 2025 models from NVIDIA, Qwen and MiniMax follow similar ratios with different cheap layers. Google's Gemma models interleave local sliding-window attention with occasional global layers in the same spirit.

#Writing more than one token per pass

The other expensive habit is decoding one token at a time. Speculative decoding already cheats by letting a small model guess ahead. Two newer ideas change the model itself.

• Multi-token prediction (MTP). Extra output heads, trained alongside the normal one, predict the tokens at $t+2$, $t+3$, … from the same hidden state (the model's internal vector at that position). Training this way tends to help the model plan ahead, and at inference the extra heads become a built-in draft model: the next pass checks their guesses and keeps the ones that match. DeepSeek-V3 trains with one extra MTP module this way.
• Diffusion language models. Instead of writing left to right, start from an answer of fixed length made entirely of [MASK] tokens. Each step predicts every masked position at once and commits the ones it is most confident about, much as image diffusion models sharpen noise into a picture. Research models such as LLaDA and commercial ones such as Inception's Mercury show this can reach competitive quality with far fewer passes.

Diffusion is not free: each pass processes the whole fixed-length answer, the length has to be chosen up front, and committing tokens in parallel can produce clashes a left-to-right model would avoid. Whether diffusion or autoregression wins at the largest scales is still an open question.

#Beyond text: images and audio as tokens

A transformer does not care what its tokens mean, only that each one arrives as a vector. That is the whole trick behind multimodal models.

• Images. A vision transformer (ViT) cuts an image into a grid of square patches, typically 14 or 16 pixels across, flattens each patch and multiplies it by a learned matrix. Each patch becomes one vector, a token like any other.
• Audio. Speech is either encoded into vectors by an audio encoder, a network that turns sound into vectors (the approach popularized by Whisper) or turned into discrete audio tokens by a neural codec, a network that compresses sound into a short sequence of codes, typically a few dozen tokens per second of sound. Codec tokens can also be generated, which is how models talk back.

There are two common ways to build such a model:

Images are expensive: one high-resolution picture can turn into thousands of tokens, which is why vision models merge or average (pool) neighboring patches before they reach the language model. That cost leads straight back to the long-context and caching chapters.

#What's next

None of these ideas replaces the core you have seen in this guide: tokens become vectors, layers mix information between positions, and training follows the gradient. They change how positions are mixed and how many tokens come out per pass. The other big frontier is not architecture at all but how models spend compute when they answer: reasoning trained with reinforcement learning, and working through long contexts full of tools, files and memory.

#Summary

• MLA caches a small latent vector per token and rebuilds per-head keys and values from it, cutting the KV cache far below GQA.
• State-space models and linear attention replace the growing cache with a fixed-size state: constant memory per token, but weaker exact recall.
• Hybrids keep a few full-attention layers among many cheap ones and get most of both worlds.
• Multi-token prediction and diffusion write several tokens per forward pass.
• Multimodal models turn image patches and audio into tokens, so the same transformer reads them all.