#From base model to assistant

A model fresh out of pretraining, the first and by far largest stage of training (next-token prediction on huge amounts of raw text, chapter 7), is a brilliant mimic and a terrible assistant. Ask it a question and it may answer, or it may write three more questions, because that is what the page it imagines it is reading would contain. Everything in this chapter is about closing that gap with surprisingly little extra training, and then giving the model abilities that do not fit inside its weights: time to think, documents to read, and tools to use.

If you are starting here, two facts from earlier chapters are enough. A language model reads text as tokens (words or pieces of words) and, given the tokens so far, gives a probability to every possible next token (chapter 1). And training means nudging the model's weights, step by step, to lower a loss: one number that measures how wrong its predictions are (chapter 7). Every stage below is that same loop with different data or a different loss.

#The training pipeline

Modern assistants go through several stages. The first uses almost all of the compute; the later ones use a sliver of it but decide how the model behaves.

#Supervised fine-tuning

Supervised fine-tuning (SFT), also called instruction tuning, is plain next-token training on example conversations. The model only ever sees one stream of tokens, so who is speaking has to be marked inside the stream. Each conversation is written out with special marker tokens, a chat template:

<|system|>You are a helpful assistant.<|end|>
<|user|>How do I boil an egg?<|end|>
<|assistant|>Lower the egg into boiling water…<|end|>

When the assistant is used, the program writes the template up to <|assistant|> and lets the model continue; the model stops when it writes <|end|>. During SFT the loss is usually computed only on the assistant's tokens: the model learns to write answers, not to predict users. A few tens of thousands of high-quality examples already change behavior dramatically, which suggests SFT mostly unlocks abilities the base model already has rather than teaching new ones.

#Preference tuning: RLHF

SFT can only imitate. Someone has to write the ideal answer, and the loss cannot say that one decent answer is better than another decent one. Yet it is much easier to judge an answer than to write a perfect one. Reinforcement learning from human feedback (RLHF) uses that asymmetry in three steps:

1. Collect comparisons. The model writes two answers to the same prompt, and a person picks the better one.
2. Train a reward model to predict those picks: it reads a prompt and an answer and outputs one number, the reward. It is usually a copy of the LLM with a single-number output.
3. Optimize the LLM against the reward model with reinforcement learning (RL), learning by trial and score: the model tries answers, the reward model scores them, and training nudges the model toward the high-scoring ones.

Step 2 uses the Bradley–Terry model of pairwise preference: the bigger the gap between two rewards, the surer the model is that people prefer the higher one:

Here $y_w$ is the preferred ("winning") answer, $y_l$ the other, and $\sigma$ the sigmoid, $\sigma(x) = 1/(1+e^{-x})$, which squashes any number into a probability between 0 and 1. For example, if the reward model gives the preferred answer 2.0 and the other 0.5, it predicts $\sigma(1.5) \approx 0.82$, an 82% chance that the person prefers the first, and the loss is $-\log 0.82 \approx 0.2$: small, little to correct. Had it scored them the other way round, $\sigma(-1.5) \approx 0.18$ and the loss is about 1.7: a big loss, so a big correction. Try being the human labeler:

Then comes step 3: the LLM itself is trained with reinforcement learning (classically PPO, proximal policy optimization, which takes small, clipped steps) to produce answers the reward model scores highly, with a penalty for drifting too far from the SFT model $\pi_\text{ref}$. The model being trained is called the policy $\pi$, a rule for picking answers; $\mathbb{E}$ averages over the answers it samples, KL measures how far it has drifted, and $\beta$ sets how strongly drifting is punished:

The KL penalty matters. A reward model is only an approximation of human taste, and a policy optimized hard against it finds its blind spots: answers that score well without being good. This is called reward hacking, and it is the reason RLHF needs careful tuning.

#DPO: skip the reward model

RLHF needs a reward model, a sampling loop and RL, and each piece can go wrong. Direct preference optimization (DPO) (Rafailov et al., 2023) gets rid of most of it with a piece of algebra. For the objective above, the best possible policy can be written as an exact formula. Turned around, that formula says the reward of an answer equals $\beta$ times the log of how much more likely the policy makes it than the reference model does, plus a term that depends only on the prompt. Put that reward into the Bradley–Terry loss: the prompt-only term cancels in the difference, and what is left is a loss on the policy's own probabilities, with no reward model in it:

Read it as: raise the probability of the preferred answer and lower the probability of the rejected one, each measured relative to the reference model. No sampling, no separate reward model, no RL loop: just supervised training on pairs. DPO and its many variants are widely used, especially by open models.

#AI feedback and constitutions

Human labels are slow and expensive, and people disagree. In RLAIF (reinforcement learning from AI feedback) another model does the labeling: it reads two answers and says which is better. Anthropic's Constitutional AI makes the judge's criteria explicit as a short list of written principles, the "constitution"; the model critiques and revises its own answers against those principles, and the AI-labeled preferences then train the reward model. The pipeline is the same as RLHF; only the source of the preference labels changes, which makes it cheap to scale and easy to inspect what the model is being taught to value.

#Reasoning: RL with verifiable rewards

For math, code and logic there is a better signal than human taste: whether the answer is right. A unit test (a small automatic check that a piece of code works) passes or it does not; a number matches the reference or it does not. Reinforcement learning with verifiable rewards (RLVR) samples many attempts per problem, rewards the ones that end with a correct, checkable answer, and updates the model toward them.

Nobody tells the model how to reason. Given room to write before answering, models trained this way learn on their own to break problems into steps, check intermediate results, notice mistakes and backtrack. Their "thinking" grows longer as training goes on. These are the reasoning models: they spend extra tokens, and therefore extra compute, at answer time.

#GRPO: the group is the baseline

A reward on its own is hard to read: 0.6 is a fine score on a hard problem and a poor one on an easy problem. What tells the model which way to move is whether an answer did better or worse than its usual attempt at that problem, so each reward is compared with a baseline. Classic PPO trains a second network, a value model, to estimate that baseline. GRPO (group relative policy optimization), used to train DeepSeek-R1, drops the value model. For each problem it samples a group of $G$ answers, scores them, and uses the group itself as the baseline:

$A_i$ is the advantage of answer $i$: how far its reward is above the group's average, divided by how spread out the rewards are (their standard deviation, std). Every token of answer $i$ is then pushed up if $A_i > 0$ and down if $A_i < 0$, with PPO's clipping and a KL penalty to the reference model keeping each step small. For example, with $G = 4$ answers rewarded 1, 0, 0, 1 (right, wrong, wrong, right), the mean is 0.5 and the std is 0.5, so the advantages are +1, −1, −1, +1: the two right answers become more likely, the two wrong ones less.

Try it on a toy policy that chooses between four ways of answering 13 × 17. The checker reads the final answer from \boxed{…}, a marker math datasets use to show where the answer is:

Two lessons hide in that figure:

• No variation, no signal. If all answers in a group get the same reward (all right or all wrong), the mean equals every reward, so every advantage is zero and the step teaches nothing. Training data is therefore chosen to be hard but not impossible for the current model.
• Reward hacking. The policy optimizes the reward it is given, not the one you meant. Real cases include models that edit unit tests to pass, skip the hard part of a task, or produce long, confident-sounding answers that fool a judge model. Careful verifiers, held-out checks, and limits on how far the policy may drift are the defenses.

#Distilling reasoning

RL on a large model is expensive. A cheaper way to give a small model reasoning ability is distillation (chapter 13): let a strong reasoning model solve many problems, keep the traces whose final answers check out, and fine-tune the small model on them with ordinary supervised learning. The same idea, generating training data with a model and filtering it automatically, is now a large share of post-training data (the data for every stage after pretraining) in general, called synthetic data.

#Test-time compute

Thinking longer is one way to spend more compute per question. Another is to sample several independent answers and take a majority vote (also called self-consistency). Both trade speed and money for accuracy. Try it on a famous trick question:

The lesson generalizes: voting and search amplify whatever the model already does most often. Test-time compute pays off on top of a model whose reasoning is mostly right, which is exactly what RL on verifiable tasks produces. More elaborate schemes score candidate answers or reasoning steps with a separate verifier or reward model and keep the best one.

#Retrieval-augmented generation

A model knows only what was in its training data, frozen at its training cutoff, and it may invent plausible details for things it half-remembers. Retrieval-augmented generation (RAG) looks the facts up first:

1. Split your documents into chunks and embed each chunk as a vector (the same idea as chapter 3, with a model trained for retrieval).
2. Embed the user's question the same way.
3. Find the chunks whose vectors are closest to the question, usually by cosine similarity, a score of how closely two vectors point the same way.
4. Paste those chunks into the prompt and ask the model to answer from them.

RAG keeps answers current and traceable (you can show the sources), and it works on private data the model never saw. Its weak point is retrieval itself: if the right chunk is not found, the model cannot use it. Production systems combine vector search with keyword search and a reranker model that re-scores the found chunks and reorders them, and long context windows (chapter 12) let them include more, larger chunks.

#Tools and agents

Some things are better done outside the model: arithmetic, looking up today's weather, running code, querying a database. Models are fine-tuned to emit tool calls, structured text such as JSON, that the surrounding program executes. The result goes back into the context and the model continues.

An agent is this loop with more tools and more steps: read files, run tests, browse, edit, and decide when the task is done. Standards such as the Model Context Protocol (MCP) describe tools in a common format so any model can use any tool server.

#Context engineering

A long-running agent's context fills up with tool results, file contents and old turns, and models get worse as their context grows long and cluttered (chapter 12). Much of building a good agent is therefore deciding what the model sees at each step: retrieving only the relevant files, trimming or summarizing old tool output, writing notes to a file and reading them back later, and handing self-contained subtasks to sub-agents with a fresh context. This is often called context engineering.

Everything rests on the same mechanics as the rest of this guide: tokens in, one predicted token out, repeated.

#Summary

• Pretraining produces a base model that continues text. Supervised fine-tuning on conversations turns it into an assistant.
• RLHF trains a reward model on human preference pairs and optimizes the LLM against it with a KL leash; DPO gets a similar result with a single supervised loss on the pairs.
• RLAIF and constitutional approaches replace human preference labels with a model judging against written principles.
• RL with verifiable rewards teaches models to reason; GRPO scores each answer against its own group, and a flawed verifier invites reward hacking.
• Distillation and synthetic data pass a big model's abilities, including reasoning, to smaller ones.
• Test-time compute (longer thinking, voting, verifiers) buys accuracy with extra tokens, and works best when the model is usually right.
• RAG puts retrieved documents into the prompt; tool use and agents let the model act through a runtime that executes its calls, and context engineering keeps what they see small and relevant.