#Mixture of experts, quantization, LoRA

Bigger models are better models, up to a point that nobody has found yet. But every parameter costs memory to store, compute to run, and money to train. Four ideas let today's models grow without the bill growing at the same rate:

• Mixture of experts: have lots of parameters, but use only a few of them for each token.
• Quantization: store each parameter in 4 or 8 bits instead of 16.
• LoRA: to adapt a model, train a tiny add-on instead of the whole thing.
• Distillation: train a small model to imitate a big one.

Other savings live elsewhere in this guide: faster decoding with the KV cache and speculative decoding in chapter 10, cache reuse in chapter 11, and architectural changes such as multi-head latent attention and multi-token prediction in chapter 14.

#Mixture of experts

More parameters let a model store more knowledge, but in a normal transformer every token pays for every parameter. Most of them sit in the feed-forward network (the MLP from chapter 6), about two thirds in large models, and every token goes through all of it. Mixture of experts breaks that link between size and cost.

A mixture-of-experts (MoE) block replaces the one MLP with many smaller ones, the experts, plus a tiny router: a small learned matrix that turns the token's vector into one score per expert. For each token, the router keeps the top few experts (often 2, sometimes 8 out of hundreds) and runs only those. The token's output is the chosen experts' outputs mixed by the router's weights:

Here $E_i(x)$ is what expert $i$ outputs for the token's vector $x$, and $g_i(x)$ is that expert's mixing weight. If the router's two best experts are 3 and 6 with weights 0.7 and 0.3, the output is 0.7 × (expert 3's output) + 0.3 × (expert 6's output), and the other experts never run for this token.

The win is the gap between total and active parameters. Mixtral 8x7B holds about 47 billion parameters but uses about 13 billion per token; DeepSeek-V3 holds 671 billion and activates about 37 billion. The model knows as much as a big model but runs at the speed of a much smaller one. The cost is memory: all experts must sit in GPU memory even though most are idle at any moment.

Three practical details:

• Load balancing. Left alone, the router tends to favor a few experts, which then train more, get better, and get picked even more. Training adds an auxiliary loss, an extra term added to the main training loss (or, in some recent models, a per-expert bias the trainer adjusts) that pushes the router to spread tokens evenly.
• Experts are not topics. It is tempting to imagine a "math expert" and a "Python expert". In practice experts specialize in fuzzier patterns, often at the level of syntax and token types, and the same expert serves many subjects.
• Fine-grained and shared experts. Recent MoE models use many small experts (dozens to hundreds) and pick several per token, plus one or two shared experts that every token goes through to hold common knowledge.

#Quantization

A model's weights are usually trained in 16-bit floating point: 2 bytes per parameter, so a 7-billion-parameter model needs about 14 GB just to hold its weights. That hurts twice. It does not fit on a typical 8 to 12 GB consumer graphics card. And it is slow: generating each token reads every weight from memory, so speed is set by how many bytes have to move (chapter 10). Quantization stores each weight with fewer bits. At 4 bits instead of 16 the model takes a quarter of the memory, and each token has a quarter of the bytes to read.

The simplest scheme, absmax quantization, takes a group of weights, finds the largest magnitude, and maps the range onto small integers:

Here $b$ is the number of bits. With $b = 4$ the integers run from −7 to +7 (one of the 16 bit patterns is left unused so the grid is symmetric around 0). The integers $q_i$ are stored, plus one scale per group. At inference time the model computes with $\hat w_i$.

A small example. Take the group 0.5, −1.2, 0.3, 2.0 and $b = 4$. The largest magnitude is 2.0, so the scale is 2.0 / 7 ≈ 0.286. Dividing by it and rounding gives the integers 2, −4, 1, 7; multiplying back gives 0.57, −1.14, 0.29, 2.0. No weight moved by more than 0.07. Now add one large weight, 6.0, to the same group: the scale triples to about 0.86, and 0.3 rounds to 0. One outlier has wiped out a small weight, which is exactly what the figure's group 3 shows.

What the figure shows is the whole story of practical quantization:

• 8 bits is close to free: quality barely changes.
• 4 bits is the popular sweet spot for running models locally; with good methods the quality loss is small.
• 3 and 2 bits start to hurt, unless the model was trained with quantization in mind.
• Outliers are the enemy. Real LLMs have a few weights and activations far larger than the rest. Small groups (32 to 128 weights per scale), keeping outliers in higher precision, or rotating the weights before quantizing (to spread outliers out) all fight the same problem.

Methods such as GPTQ and AWQ choose the rounding more cleverly, using a little calibration data to decide which weights matter most. Formats like GGUF (the file format of llama.cpp, a popular program for running models on ordinary computers) and the 8-bit and 4-bit floating-point formats on newer GPUs (FP8, FP4) are variations on the same idea. Quantization is also creeping into training: DeepSeek-V3, for example, ran most of its training matrix multiplications in 8-bit floating point (FP8) to save time and memory.

#LoRA: fine-tuning on a budget

Fine-tuning means continuing to train a model on your own data, for example your company's support conversations. Updating every weight needs memory for the weights, their gradients, and the optimizer's running averages for each one (chapter 7). With the usual Adam optimizer and mixed precision that is about 16 bytes per parameter: over 100 GB for a 7B model, far more than one ordinary GPU holds.

LoRA (low-rank adaptation) freezes the original weight matrix $W$ and learns a small correction as the product of two thin matrices:

Here $B$ is a table of $d$ by $r$ numbers, $A$ one of $r$ by $d$, and $r \ll d$ means $r$ is much smaller than $d$. The number $r$ is the rank. To see what it buys, take $r = 1$: $B$ is a single column and $A$ a single row, and their product is a full table in which every row is a multiple of the same row:

Six numbers describe nine. For a real layer with $d = 4096$, rank 1 uses 8,192 numbers to describe a change to 16.8 million; rank $r$ adds up $r$ such tables. The bet is that the change fine-tuning needs is low rank: it can be described by a few directions instead of $d^2$ independent numbers. That bet works remarkably well.

In practice:

• Typical ranks are 8 to 64, so LoRA trains well under 1% of the parameters.
• $B$ starts at zero, so $BA = 0$ and training begins exactly at the original model.
• After training, $BA$ can be added into $W$: no extra cost at inference. Or adapters can be kept separate and swapped per customer or per task on one shared base model.
• QLoRA combines two ideas of this chapter: the frozen base model is stored in 4 bits, and only the small LoRA matrices are trained in higher precision. That puts fine-tuning of fairly large models within reach of a single GPU.

#Distillation: a big model teaches a small one

A trained model knows more than which token is right. When it predicts the next word after "The cat sat on the", it gives "mat" most of the probability but also says that "rug" and "floor" are reasonable and "banana" is absurd. Knowledge distillation trains a small student on these full distributions from a large teacher, instead of on the one correct token alone (Hinton et al., 2015):

Here $z$ are the raw scores (logits) each model gives every possible token before softmax. KL is the Kullback–Leibler divergence, a measure of how different two probability distributions are: 0 when they are identical, larger the more they disagree. The temperature $T$ (the same knob as in chapter 9) flattens both distributions so the small probabilities, the "dark knowledge", carry weight in the loss. The factor $T^2$ only keeps the size of the training signal about the same whichever $T$ you pick.

Distillation is how most small, capable models are made today. Sometimes the student matches the teacher's probabilities directly (this needs access to the teacher's logits, which closed models usually do not share); often it simply trains on text the teacher wrote, which is called synthetic data or sequence-level distillation. Small reasoning models are commonly distilled from the long reasoning traces of larger ones (chapter 15).

#Pruning

The bluntest saving: if parts of a trained model barely matter, cut them out. Remove weights, neurons, attention heads, or whole layers that contribute little, then briefly retrain to recover. Structured pruning (removing whole pieces) gives real speedups; scattering zeros through a matrix usually does not, unless the hardware supports that sparsity pattern. Pruning and distillation are often combined: prune a big model into a smaller shape, then distill the original into it.

#Summary

• MoE separates what a model knows (total parameters) from what each token costs (active parameters).
• Quantization stores weights as small integers plus a scale; 8 bits is nearly lossless, 4 bits is the common sweet spot, outliers are the main difficulty.
• LoRA freezes the model and trains a low-rank correction $BA$; QLoRA does that on a 4-bit base model.
• Distillation trains a small student on a big teacher's full probability distributions or on its generated text.
• Pruning cuts out the parts that contribute least; removing whole pieces gives real speedups.

Next: new layer designs that go beyond the classic transformer, in chapter 14.