Sampling

Training is done: the model now puts high probability on plausible next tokens. Using it means running it forward over and over, adding one token each time (chapter 1). At the end of every forward pass the model hands over one number per vocabulary entry: a logit. It does not choose a token. Choosing is a separate, small piece of code called the sampler, and its settings (the decoding strategy) change the character of the text more than most people expect. The same model can be dull, creative or incoherent depending on a few knobs.

The obvious rule, always take the most likely token, turns out to be a poor one (Figure 9.3 shows why). So samplers draw at random, in proportion to the probabilities, and shape those probabilities first.

From logits to probabilities

Softmax turns logits $z_i$ into probabilities. The temperature $T$ divides every logit before the softmax:

p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

$T = 1$ leaves the model's distribution unchanged.
$T < 1$ stretches the gaps between logits, so the favorite gets even more likely. As $T \to 0$ this becomes greedy decoding: always take the top token.
$T > 1$ shrinks the gaps, so unlikely tokens catch up. At very high $T$ every token is nearly equally likely.

Take the three scores from chapter 1, 2, 1 and 0, which softmax turns into 67%, 24% and 9%. At $T = 0.5$ they become 4, 2 and 0, and the probabilities become 87%, 12% and 2%. At $T = 2$ they become 1, 0.5 and 0, and the probabilities 51%, 31% and 19%. Temperature never changes the order of the tokens, only how far apart their chances are.

Figure 9.1 · Temperature reshapes the same logits

My favorite food is ▍

·pizza·sushi·pasta·chocolate

logit÷ Ttokenprobability

4.14.1·pizza39.3%

3.63.6·sushi23.8%

3.23.2·pasta16.0%

2.62.6·chocolate8.8%

2.32.3·tacos6.5%

1.51.5·anything2.9%

0.90.9·my1.6%

0.20.2·cold0.8%

−0.8−0.8·purple0.3%

−1.6−1.6·Tuesday0.1%

The logits on the left are fixed: they are what the model computed for 'My favorite food is'. Drag the temperature and watch the probabilities; the strip on top is the whole distribution at once. 'Effective choices' is e raised to the entropy (the average of −ln p, weighted by each token's probability; a measure of uncertainty): roughly how many tokens are realistically in play.

Cutting off the long tail

A real vocabulary has around 200,000 tokens. Even when each nonsense token has a tiny probability, together they add up, and sooner or later one gets drawn. One bad token can derail everything after it, because the model now has to continue from its own mistake. Samplers therefore truncate the distribution before drawing:

Top-k keeps only the $k$ most likely tokens.
Top-p (nucleus sampling) keeps the smallest set of top tokens whose probabilities add up to at least $p$ . When the model is confident that set is tiny; when it is unsure, it grows.
Min-p keeps tokens whose probability is at least $p_{\min}$ times the top token's probability, so the cutoff scales with confidence.

After cutting, the survivors are renormalized to sum to 1 again, and one token is drawn at random in proportion to its probability.

In the lab below, at $T = 1$ , the top five tokens have 39%, 24%, 16%, 9% and 6.5%. Top-k of 3 keeps pizza, sushi and pasta. Top-p of 0.9 keeps five: the first four add up to only 88%, so tacos joins and the total reaches 94%. Min-p of 0.1 drops everything below 3.9% (a tenth of pizza's 39%), which also leaves five. Now raise the temperature: top-k still keeps three, while top-p and min-p let more tokens in as the model's confidence spreads out.

Figure 9.2 · A sampling lab

Temperature1.00Top-koffTop-poffMin-poff

tokenprobabilitydraws

·pizza39.3%0

·sushi23.8%0

·pasta16.0%0

·chocolate8.8%0

·tacos6.5%0

·anything2.9%0

·my1.6%0

·cold0.8%0

·purple0.3%0

·Tuesday0.1%0

Latest draws

Struck-out tokens can no longer be picked. Press 'Sample 100 times' to draw real tokens: with only 100 draws the histogram is noisy, but it tracks the probabilities. Each press uses a new seed (the starting number of the random generator).

Inference engines (the software that runs trained models for users) typically apply temperature first, then the filters. Common defaults are around $T = 0.7$ to $1$ with top-p between $0.9$ and $0.95$ , but every provider tunes these differently.

What the knobs do to text

Run the same prompt through a toy language model, a table of which word tends to follow which, at four temperatures.

Figure 9.3 · One prompt, four temperatures

T = 0 (greedy)

Once upon a time, there was a cat who liked to sleep in the sun and the sun and

T = 0.7

Once upon a time there lived a happy dog who loved the garden. Then the sun. It was

T = 1.2

Once upon a time there lived a was a dog that loudly the sun. cat. Then a dog

T = 2.5

Once upon a time there tax moon rain purple sun It liked the zebra purple rain Tuesday Then loved the

Greedy decoding falls into a loop ('the sun and the sun and…'), a classic failure. Moderate temperatures read naturally. High temperature lets the long tail in and produces word salad. Turn on the repetition penalty to see the greedy loop break.

Greedy decoding is not "the most accurate" text; it is the text that never takes a risk, and language models that never take risks tend to repeat themselves. Some randomness is what makes the output read like language.

Structured output: sampling under a grammar

Programs that call a model often need output in a fixed format: JSON (a common text format for structured data) that matches a schema (the required fields and their types), a function call, a single label from a list. Asking politely in the prompt works most of the time. Constrained decoding makes it work every time.

The trick happens between the logits and the draw. A small parser, a program that checks text against a format's rules (its grammar), tracks where we are in the grammar (inside a string, expecting a key, expecting a digit…) and knows which tokens could legally come next. Every other token gets its logit set to $-\infty$ , and since $e^{-\infty} = 0$ , softmax gives it exactly zero probability. The model still chooses among the valid tokens by its own preferences.

Figure 9.4 · Constrained decoding: a JSON grammar masks the logits

Output▍

Grammar expects the start of an object: only "{" fits

tokenlogitmaskedprobability

Sure3.1·45%

Here2.4·22%

{2.0·15%

```json1.8·12%

OK1.0·6%

1/19

The model's raw scores. Its favorite is Sure, which would break the JSON.

The schema asks for {'name': string, 'age': integer}. At each step you first see the model's raw scores, then the grammar's mask: struck-out tokens get −∞ and drop to 0%. Watch the model's favorites 'Sure', an unquoted Ada and the string '36' get vetoed.

Real engines compile the JSON schema or grammar into a state machine (a precomputed table of which tokens are allowed in each state) over the tokenizer's vocabulary ahead of time, so the mask for each step costs microseconds. The same mechanism powers "JSON mode", tool calls and outputs constrained by a regex (a pattern language for text) in most serving stacks, the software that runs models for many users at once.

A mask guarantees form, not truth

A grammar can force "age": 36 to be an integer, but not the right integer. Masking can also push the model into a corner it would not have chosen, so a schema that fights the model's natural phrasing can lower answer quality.

A few more tools

Repetition and frequency penalties lower the logits of tokens that already appeared, discouraging loops like the one above. They have to be gentle, because some repetition (the, names, code) is correct.
Seeds. The "randomness" comes from a pseudo-random generator. Fix its seed and the same settings reproduce the same text, which is handy for testing. (In practice tiny numerical differences on GPUs can still make outputs drift.)
Logprobs. Most APIs (the interfaces programs use to send requests to a hosted model) can return the log-probability of every chosen token, and of the top few alternatives. They are the model's own confidence, useful for scoring answers, spotting guesses and building classifiers out of a chat model.
Beam search keeps several candidate continuations in parallel and picks the one with the best total probability. It helps translation, but for open-ended chat it makes text bland, so chat models rarely use it.

Temperature 0 for facts, higher for ideas

For extraction, classification or code where there is one right answer, low temperature (or greedy) is usually best. For brainstorming and fiction, a higher temperature gives more variety. The model is the same; only the sampler changes.

Summary

The model outputs logits; the sampler turns them into one chosen token.
Temperature sharpens ( $T < 1$ ) or flattens ( $T > 1$ ) the distribution without reordering it.
Top-k, top-p and min-p cut off the unreliable tail before drawing.
Greedy decoding tends to loop; a bit of randomness makes text natural.
Constrained decoding masks invalid tokens with $-\infty$ , so structured output always parses.

Sampling is cheap. Producing the logits is the expensive part, and the next chapter shows how inference engines make that fast.

#Sampling

#From logits to probabilities

#Cutting off the long tail

#What the knobs do to text

#Structured output: sampling under a grammar

#A few more tools

#Summary