I9 · Generative AI and Large Language Models in Practice

Sampling and Temperature

At each step an LLM outputs a probability for every token in its vocabulary. Decoding, or sampling, chooses the next token from this distribution. Greedy decoding always takes the most probable token. Temperature divides the model's scores (logits) before softmax: below 1 it sharpens the distribution, making output more predictable; above 1 it flattens it, making output more varied. Top-k sampling keeps only the k most probable tokens; top-p, or nucleus, sampling keeps the smallest set whose probabilities add up to p. A random seed makes sampling repeatable.

8 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Ask the same chatbot for a story twice and you get two different stories. Ask it for the sum of two numbers and you want the same answer every time. A few settings, temperature, top-k and top-p, decide how adventurous the model is when picking each next word.

At each step an LLM outputs a probability for every token in its vocabulary. Decoding, or sampling, chooses the next token from this distribution. Greedy decoding always takes the most probable token. Temperature divides the model's scores (logits) before softmax: below 1 it sharpens the distribution, making output more predictable; above 1 it flattens it, making output more varied. Top-k sampling keeps only the k most probable tokens; top-p, or nucleus, sampling keeps the smallest set whose probabilities add up to p. A random seed makes sampling repeatable.

In simple words

Imagine choosing tonight's dinner from a menu where each dish has a popularity score. At low temperature you nearly always pick the favourite, dal chawal. At high temperature you might pick something surprising. Top-k says "only consider the top five dishes".

Overview

  • At each step an LLM outputs a probability for every possible next token.
  • Decoding is the method of choosing the next token from those probabilities.
  • Temperature rescales probabilities: low values make output focused, high values make it varied.
  • Top-k and top-p (nucleus) sampling limit choices to the most likely tokens.
  • Low temperature suits facts and structured output; higher suits brainstorming and creative writing.
  • Even at temperature 0, outputs may vary slightly between runs on some systems.

Why it matters

Sampling settings change the character of a model's output as much as the prompt does. The right settings make answers consistent for data extraction and grading, and varied for brainstorming and creative writing.

Key terms

Logits
The raw scores the model gives each possible next token before softmax.
Greedy decoding
Always choosing the single most probable next token.
Temperature
A number that sharpens (low) or flattens (high) the probability distribution.
Top-k sampling
Sampling only from the k most probable tokens.
Top-p (nucleus) sampling
Sampling from the smallest set of tokens whose probabilities sum to at least p.
Seed
A starting value for the random number generator that makes sampling repeatable.

Types

There are several decoding strategies. Each trades predictability against variety.

1. Greedy decoding

Always choosing the single most likely next token.

  • How it works: It is equivalent to temperature 0: fast and repeatable, but it can produce repetitive, dull text.
  • Example: Extracting a date from an invoice, where creativity is unwanted.
  • When to use it: Use it for classification, extraction and other tasks with one correct answer.

2. Temperature sampling

Sampling from probabilities that have been sharpened or flattened by a temperature T.

  • How it works: Logits are divided by T before the softmax; T < 1 favours likely tokens, T > 1 spreads probability to unlikely ones.
  • Example: At T = 0.2 a story is predictable; at T = 1.2 it becomes surprising and sometimes incoherent.
  • When to use it: Use about 0.0–0.3 for facts and code and 0.7–1.0 for creative or varied outputs.

3. Top-k sampling

Sampling only from the k most likely tokens.

  • How it works: Tokens outside the top k get zero probability; the rest are renormalised before sampling.
  • Example: With k = 40, very unlikely, nonsensical words are never chosen.
  • When to use it: Use it to avoid rare, odd tokens while keeping some variety. A fixed k can be too strict when many words fit and too loose when only one does.

4. Top-p (nucleus) sampling

Sampling from the smallest set of tokens whose probabilities add up to p.

  • How it works: The set adapts: when the model is confident, few tokens qualify; when unsure, more do (Holtzman et al., 2019).
  • Example: With p = 0.9, the model chooses among tokens covering 90% of the probability.
  • When to use it: Use it as a common default with temperature for natural-sounding text.

5. Beam search and other controls

Keeping several candidate sequences at once, plus penalties and stop sequences.

  • How it works: Beam search keeps the k best partial outputs; repetition and frequency penalties discourage repeats; stop sequences end generation at chosen text.
  • Example: Machine translation systems using beam search with width 4 or 5.
  • When to use it: Use beam search for translation and summarisation, and penalties when outputs repeat themselves.

Explained in detail

How temperature works

Suppose after "I drank a cup of" the model's logits for "tea", "coffee", "water" and "ink" are 3.0, 2.0, 1.0 and −1.0. Softmax turns them into probabilities of about 0.66, 0.24, 0.09 and 0.01. With temperature 0.5, each logit is divided by 0.5 before softmax, doubling the gaps, and "tea" rises to about 0.86. With temperature 1.5, the gaps shrink and "tea" falls to about 0.54 while even "ink" gets a small chance. Temperature near 0 approaches greedy decoding. Because each sampled token becomes part of the context for the next one, a single unusual choice can send the whole continuation in a new direction, which is why high temperatures can drift into nonsense.

Choosing settings by task

TaskSuggested settingsWhy
Extracting fields, classification, gradingTemperature 0 to 0.2Consistency matters most
Factual questions, summaries, codeTemperature about 0.2 to 0.5Mostly reliable with some fluency
Conversation and explanationsTemperature about 0.7, top-p about 0.9Natural, not repetitive
Brainstorming, stories, slogansTemperature about 0.9 to 1.2Variety and surprise

Top-p sampling, introduced by Ari Holtzman and colleagues in 2019 as nucleus sampling, adapts to the situation: when the model is confident, the nucleus may contain only one or two tokens; when many continuations are plausible, it contains many. This avoids both the dull, repetitive text that greedy decoding often produces and the strange words that come from sampling the long tail of unlikely tokens. Most providers expose temperature and top-p; changing one at a time is easier to reason about. Even at temperature 0, outputs can occasionally differ slightly between runs because of how computations are spread across hardware, so do not rely on exact repetition for critical systems; validate outputs instead.

Repetition is a common failure at low temperature with some models: the same phrase loops again and again. Providers offer frequency or presence penalties that lower the scores of tokens already used, which reduces loops. Another useful pattern is self-consistency: sample several answers to a reasoning question at a moderate temperature and take the answer that appears most often, which is usually more accurate than a single greedy answer. These settings are part of the system's design and should be recorded alongside the prompt, so that results can be reproduced and compared when the prompt or model changes.

How it works

  1. Decide whether the task needs consistency or variety.
  2. Set temperature low for extraction and grading, higher for creative work.
  3. Adjust top-p only if outputs are too dull or too strange.
  4. Generate several samples and compare them on the same prompt.
  5. Validate important outputs instead of trusting a single sample.

In code

sampling.py
import numpy as np

tokens = ["tea", "coffee", "water", "juice", "ink"]
logits = np.array([3.0, 2.0, 1.0, 0.5, -1.0])

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

for t in [0.5, 1.0, 1.5]:
    print(f"temperature {t}:", {w: round(float(p), 2) for w, p in zip(tokens, softmax(logits / t))})

def top_k(probs, k):
    keep = np.argsort(probs)[::-1][:k]
    out = np.zeros_like(probs); out[keep] = probs[keep]
    return out / out.sum()

def top_p(probs, p):
    order = np.argsort(probs)[::-1]
    cutoff = np.searchsorted(np.cumsum(probs[order]), p) + 1
    out = np.zeros_like(probs); out[order[:cutoff]] = probs[order[:cutoff]]
    return out / out.sum()

base = softmax(logits)
print("top-k=2 keeps:", [w for w, p in zip(tokens, top_k(base, 2)) if p > 0])
print("top-p=0.9 keeps:", [w for w, p in zip(tokens, top_p(base, 0.9)) if p > 0])

rng = np.random.default_rng(42)
for t in [0.2, 1.5]:
    picks = rng.choice(tokens, size=200, p=softmax(logits / t))
    counts = {w: int((picks == w).sum()) for w in tokens}
    print(f"200 samples at temperature {t}: {counts}")

Output

temperature 0.5: {'tea': 0.86, 'coffee': 0.12, 'water': 0.02, 'juice': 0.01, 'ink': 0.0}
temperature 1.0: {'tea': 0.62, 'coffee': 0.23, 'water': 0.08, 'juice': 0.05, 'ink': 0.01}
temperature 1.5: {'tea': 0.49, 'coffee': 0.25, 'water': 0.13, 'juice': 0.09, 'ink': 0.03}
top-k=2 keeps: ['tea', 'coffee']
top-p=0.9 keeps: ['tea', 'coffee', 'water']
200 samples at temperature 0.2: {'tea': 200, 'coffee': 0, 'water': 0, 'juice': 0, 'ink': 0}
200 samples at temperature 1.5: {'tea': 97, 'coffee': 50, 'water': 31, 'juice': 18, 'ink': 4}

Try it yourself

Two jobs, two settings

A startup uses one LLM for two jobs: reading invoices and writing ad slogans.

Low or high temperature?

Pick a setting for each task.

  • Mark answers against a rubric

  • Extract the date and amount from a receipt

  • Suggest 15 names for a school science fair

  • Write a playful poem about monsoon

Common mistakes

  • Using a high temperature for data extraction or grading.
  • Changing temperature and top-p at the same time and not knowing which mattered.
  • Assuming temperature 0 guarantees identical, correct answers.

Check yourself

Question 1

1. Lowering the temperature makes the output…

Question 2

2. Top-p = 0.9 keeps…

Question 3

3. Which task suits a high temperature?

Summary

  • The model gives a probability for every next token; decoding chooses one.
  • Temperature sharpens or flattens choices; top-k and top-p remove unlikely tokens.
  • Use low temperature for consistency and higher for creativity, and validate important outputs.

Finished reading?

Sources: Holtzman et al. (2019), The Curious Case of Neural Text Degeneration (nucleus sampling) · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft): Large Language Models