Definition
A model that has read a large part of the public internet can complete sentences brilliantly but may ignore your question or say something harmful. Turning that raw text predictor into a helpful assistant takes several stages of training.
Modern LLMs are trained in stages. In pre-training, a transformer learns to predict the next token on trillions of tokens of text, which teaches grammar, facts and patterns of reasoning; this produces a base model. In instruction tuning, or supervised fine-tuning, it is trained on examples of instructions and good responses. In preference tuning, such as reinforcement learning from human feedback (RLHF) or direct preference optimisation (DPO), it learns from human or AI comparisons of which answer is better, making it more helpful, honest and harmless. Safety testing, called red-teaming, and evaluation follow before release.
In simple words
First a student reads an entire library and learns how language works (pre-training). Then a teacher shows worked examples of good answers (instruction tuning). Finally, the teacher compares pairs of the student's answers and says which is better, again and again (preference tuning).
Overview
- LLMs are trained in stages: pre-training, supervised fine-tuning and preference alignment.
- Pre-training predicts the next token on trillions of tokens of web pages, books and code.
- It takes thousands of GPUs for weeks or months and costs millions of dollars for frontier models.
- Scaling laws show that loss falls predictably as model size, data and compute grow.
- Fine-tuning and alignment turn a text predictor into a helpful, safer assistant.
- Data quality, filtering and licensing matter as much as size.
Why it matters
Knowing how LLMs are trained explains their behaviour: why they know a lot but can be out of date, why they sound confident even when wrong, why they refuse some requests and why their abilities depend on the data they saw.
Key terms
- Pre-training
- Learning to predict the next token on a huge, general text corpus.
- Base model
- The pre-trained model before instruction tuning; it continues text rather than following instructions.
- Instruction tuning
- Fine-tuning on examples of instructions paired with good responses.
- RLHF
- Reinforcement learning from human feedback: training a reward model on human preferences and optimising the LLM against it.
- Knowledge cutoff
- The date after which the model's training data has no information.
- Red-teaming
- Deliberately trying to make a model behave badly in order to find and fix weaknesses.
Types
LLM training has several types of stage and data. Each stage teaches something different.
1. Pre-training data collection and cleaning
Gathering and filtering huge text and code collections.
- How it works: Web crawls such as Common Crawl are de-duplicated, filtered for quality and harmful content, and mixed with books, code and Wikipedia.
- Example: Removing spam pages and copies of the same article before training.
- When to use it: Understand it to see why models know some topics and languages better than others.
2. Self-supervised pre-training
Training the model to predict the next token across the whole corpus.
- How it works: Cross-entropy loss is minimised with AdamW on huge batches; the model learns grammar, facts and reasoning patterns as side effects.
- Example: GPT-3 (2020) was pre-trained with 175 billion parameters on about 300 billion tokens.
- When to use it: Recognise it as the stage that gives broad knowledge and most of the cost.
3. Supervised fine-tuning (instruction tuning)
Training on example prompts paired with high-quality responses.
- How it works: Tens of thousands of demonstrations teach the model to follow instructions, use formats and hold conversations.
- Example: Examples such as "Summarise this email in three bullet points" with an ideal answer.
- When to use it: Recognise it as the step that makes a model usable as an assistant.
4. Preference alignment (RLHF, DPO, constitutional methods)
Training the model to prefer helpful, honest and harmless responses.
- How it works: In RLHF, people rank responses, a reward model learns their preferences and reinforcement learning (often PPO) tunes the LLM; DPO learns from preferences directly; constitutional methods use written principles.
- Example: InstructGPT (2022) showed that a smaller aligned model was preferred over a much larger base model.
- When to use it: Recognise it as the stage that shapes tone, refusals and safety behaviour.
5. Continued training and evaluation
Further domain training, reasoning training and testing before release.
- How it works: Models may be trained further on code, maths or a language, taught to reason with reinforcement learning, and tested on benchmarks and red-teaming.
- Example: Red-teamers trying to make a model give dangerous instructions before launch.
- When to use it: Recognise these steps when reading model cards and safety reports.
Explained in detail
Stage 1: pre-training
Pre-training data comes from web pages, books, code, encyclopaedias and other sources, filtered to remove spam, duplicates and some harmful content. The training objective is simple: given the tokens so far, predict the next one, and adjust billions of weights by gradient descent to make the correct next token more probable. The loss is cross-entropy, the negative log probability given to the true next token. Doing this over trillions of tokens on thousands of GPUs for weeks or months teaches the model spelling, grammar, world knowledge, styles of writing and many reasoning patterns, because predicting the next word well requires them. Research on scaling laws, such as work published by OpenAI in 2020 and DeepMind's "Chinchilla" paper in 2022, showed that loss falls smoothly as model size, data and compute grow together.
Stages 2 and 3: becoming an assistant
| Stage | Data | What it changes |
|---|---|---|
| Instruction tuning (SFT) | Tens of thousands to millions of instruction-response examples, many written or checked by people | Follows instructions, answers questions, uses a chat format |
| Preference tuning (RLHF, DPO) | Pairs of answers ranked by people or by an AI using written principles | More helpful, clearer, safer; declines harmful requests |
| Safety evaluation and red-teaming | Adversarial prompts written by experts and automated tools | Finds weaknesses to fix before release |
OpenAI's InstructGPT paper of 2022 showed that a much smaller model trained with human feedback was preferred by people over a far larger base model, showing how much these later stages matter. Anthropic's Constitutional AI approach, published in 2022, uses a written set of principles so that an AI model can produce much of the preference feedback itself. Training does not stop at release: models are updated, and many products add retrieval and tools so they can use information newer than their knowledge cutoff.
- Because training data reflects the internet, models can repeat its biases and errors; filtering and preference tuning reduce but do not remove this.
- Fine-tuning by users, for example on a company's support tickets, adapts style and format; for new facts, retrieval is usually cheaper and easier to update.
Training a frontier model is extremely expensive, which is why most developers use existing models through APIs or adapt open-weight models rather than training their own.
How it works
- Collect and clean a very large text corpus.
- Pre-train a transformer to predict the next token.
- Fine-tune on instruction-response examples.
- Tune on preference comparisons with RLHF or DPO.
- Red-team, evaluate and document before release.
In code
import math
import random
from collections import Counter, defaultdict
corpus = ("the farmer waters the field . the farmer sells rice at the market . "
"the student reads a book . the student writes notes in class . "
"the farmer reads the weather report .").split()
# "Pre-training" a bigram model: count which word follows which
counts = defaultdict(Counter)
for a, b in zip(corpus, corpus[1:]):
counts[a][b] += 1
vocab = sorted(set(corpus))
def prob(a, b, smoothing=0.1):
return (counts[a][b] + smoothing) / (sum(counts[a].values()) + smoothing * len(vocab))
def avg_loss(words, model):
return sum(-math.log(model(a, b)) for a, b in zip(words, words[1:])) / (len(words) - 1)
uniform = lambda a, b: 1 / len(vocab)
test = "the farmer reads a book .".split()
print(f"Vocabulary: {len(vocab)} words")
print(f"Loss before training (uniform guess): {avg_loss(test, uniform):.2f}")
print(f"Loss after counting bigrams: {avg_loss(test, prob):.2f}")
random.seed(1)
word, out = "the", ["the"]
for _ in range(8):
nxt = random.choices(vocab, weights=[prob(word, v) for v in vocab])[0]
out.append(nxt)
word = nxt
print("Generated:", " ".join(out))
print("Most likely after 'farmer':", counts["farmer"].most_common(2))Output
Vocabulary: 19 words
Loss before training (uniform guess): 2.94
Loss after counting bigrams: 1.17
Generated: the farmer waters the farmer sells rice market rice
Most likely after 'farmer': [('waters', 1), ('sells', 1)]Try it yourself
The life of an LLM
Put the training stages in order.
- 1Preference-tune with human or AI comparisons
- 2Instruction-tune on example conversations
- 3Collect and filter a huge text corpus
- 4Pre-train by predicting the next token
- 5Release, monitor and update
- 6Red-team and evaluate for safety
Match the behaviour to the training stage
Which stage mainly explains each?
- Knows facts and grammar
- Answers in a chat format when asked
- Prefers clear, polite, safe answers
- Does not know last month's news
Common mistakes
- Believing an LLM looks facts up in a database when it answers.
- Expecting a model to know events after its knowledge cutoff.
- Fine-tuning to add facts that change often, instead of using retrieval.
Check yourself
Summary
- Pre-training predicts the next token on huge text, producing a knowledgeable base model.
- Instruction tuning and preference tuning (RLHF, DPO) make it a helpful, safer assistant.
- Training data sets the knowledge cutoff and biases; retrieval and tools add fresh information.
Finished reading?
Sources: Ouyang et al. (2022), Training language models to follow instructions with human feedback · Andrej Karpathy: Let's build GPT, from scratch, in code