I9 · Generative AI and Large Language Models in Practice

Fine-Tuning and Adapting LLMs

Fine-tuning continues training a pre-trained model on examples of a task or behaviour. In full fine-tuning all weights are eligible for updates. LoRA freezes a weight matrix W and learns two smaller matrices A and B. Their product BA has rank at most r. Writing the scaling factor as part of B gives the effective weight W + BA. QLoRA combines trainable adapters with a quantised frozen base model to reduce memory use. The memory needed still depends on the model, sequence length and training settings.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

A hospital wants an assistant that writes discharge summaries in its own format, with its own medical abbreviations. A general chatbot writes decent summaries but in the wrong style. Should the hospital write better prompts, give the model its documents, or retrain part of the model? This page explains the options and why fine-tuning has become cheap.

Fine-tuning continues training a pre-trained model on examples of a task or behaviour. In full fine-tuning all weights are eligible for updates. LoRA freezes a weight matrix W and learns two smaller matrices A and B. Their product BA has rank at most r. Writing the scaling factor as part of B gives the effective weight W + BA. QLoRA combines trainable adapters with a quantised frozen base model to reduce memory use. The memory needed still depends on the model, sequence length and training settings.

In simple words

A new doctor already knows medicine (pre-training). Giving them a checklist each day is prompting. Handing them the patient's file is RAG. Sending them on a two-week course in the hospital's procedures is fine-tuning. LoRA is like adding a thin notebook of new habits instead of rewriting their whole medical education.

Overview

  • Adapting an LLM means changing its behaviour or knowledge for a specific task or domain.
  • Options range from prompting and RAG (no training) to full fine-tuning (training all weights).
  • Fine-tuning teaches style, format and task skills; RAG is better for adding changing facts.
  • Parameter-efficient methods such as LoRA train only a small number of extra weights.
  • A few hundred to a few thousand high-quality examples are often enough.
  • Fine-tuned models must be re-evaluated for accuracy, safety and forgetting of general skills.

Why it matters

Organisations want models that follow their formats, speak in their tone, use their terminology or work in under-served languages. Choosing correctly between prompting, RAG and fine-tuning saves money and avoids common failures, such as fine-tuning to add facts that change every week.

Key terms

Pre-training
The first, huge training run on general text that gives an LLM broad knowledge.
Supervised fine-tuning (SFT)
Further training on example prompts with ideal responses.
Parameter-efficient fine-tuning (PEFT)
Training only a small set of added parameters while freezing the original model.
LoRA
Learning a low-rank update BA to frozen weight matrices.
Quantisation
Storing weights with fewer bits, such as 4 instead of 16, to save memory.
Catastrophic forgetting
Losing general abilities after training too hard on a narrow task.

Types

There are several types of LLM adaptation. They differ in cost, data needs and what they change.

1. Prompt engineering and in-context learning

Adapting behaviour only through instructions and examples in the prompt.

  • How it works: No weights change; system prompts, few-shot examples and formats steer the model each time it is called.
  • Example: A support bot told its tone, policies and three example answers in the system prompt.
  • When to use it: Use it first; it is instant, cheap and easy to change.

2. Retrieval augmentation

Supplying knowledge from documents at question time.

  • How it works: Relevant chunks are retrieved and added to the prompt, so updating knowledge means updating documents, not retraining.
  • Example: A bank assistant answering from this month's interest-rate sheet.
  • When to use it: Use it for private or frequently changing facts and when citations are needed.

3. Full fine-tuning

Continuing training of all model weights on task-specific examples.

  • How it works: Supervised training on input-output pairs updates every parameter, which needs much GPU memory and risks forgetting earlier skills.
  • Example: Training a 1-billion-parameter model to write discharge summaries in a hospital's format.
  • When to use it: Use it for smaller open models when you have lots of data and compute.

4. Parameter-efficient fine-tuning (LoRA, QLoRA, adapters)

Training small added modules while the original weights stay frozen.

  • How it works: LoRA (Hu et al., 2021) learns low-rank update matrices, often under 1% of the parameters; QLoRA also quantises the base model so it fits on one GPU.
  • Example: Fine-tuning a 7-billion-parameter model on a single consumer GPU for customer-support replies.
  • When to use it: Use it as the default way to fine-tune open models cheaply; several adapters can share one base model.

5. Preference tuning and distillation

Training on preferred versus rejected answers, or training a small model to copy a large one.

  • How it works: DPO uses pairs of good and bad responses to shift style and safety; distillation trains a student model on a teacher model's outputs.
  • Example: A small, fast model trained on 50,000 answers from a large model for a narrow task.
  • When to use it: Use them to align tone and refusals, or to cut cost and latency for a specific task, checking licence terms.

Explained in detail

Prompting, RAG or fine-tuning?

NeedBest first choiceWhy
Follow a format or tone for a few tasksPrompting with examplesCheapest and fastest to change
Answer from company documents that change oftenRAGFacts stay current and answers can cite sources
A consistent style or structure across thousands of outputsFine-tuningBehaviour is learned, so prompts can be shorter
A narrow skill, such as classifying tickets or extracting fieldsFine-tuning a smaller modelCheaper and faster at large volumes
A low-resource language or domain jargonFine-tuning, often with RAGThe model must learn new patterns of language

A common mistake is fine-tuning a model to "learn" facts such as prices or policies. Fine-tuning is good at teaching behaviour, but it stores facts unreliably, cannot cite where they came from, and needs retraining whenever they change; retrieval handles facts better. Many real systems combine all three: a fine-tuned model that knows the house style, prompted with clear instructions, and given retrieved documents for the facts.

How LoRA makes fine-tuning cheap

Full fine-tuning of a 7-billion-parameter model with the Adam optimiser needs memory for the weights, their gradients and two optimiser values each, roughly 16 bytes per parameter in mixed precision, or about 112 GB before activations and other overhead. LoRA observes that the change needed for a new task usually has low rank: it can be written as the product of a tall, thin matrix B (d × r) and a short, wide matrix A (r × d) with r as small as 4 to 16. For a 4,096 × 4,096 matrix, full fine-tuning trains 16.8 million numbers; LoRA with r = 8 trains 65,536, about 0.4 per cent. The original weights stay frozen, so many small LoRA adapters, one per customer or task, can share a single base model and be swapped in at run time. QLoRA stores the frozen base model in 4 bits (about 3.5 GB for 7 billion parameters) and trains LoRA adapters on top. The code below also builds a 256 × 256 weight change that is secretly rank 4 plus a little noise: approximating it with rank 1 or 2 leaves large errors (84 and 66 per cent), but rank 4 already brings the error down to 2.5 per cent while storing only 2,048 numbers instead of 65,536.

Data, evaluation and risks

Quality matters more than quantity: a few hundred to a few thousand carefully written examples of exactly the desired input and output often suffice for style and format. Hold out a test set, compare the fine-tuned model with a well-prompted base model, and check general abilities too, because heavy fine-tuning on a narrow task can cause catastrophic forgetting. Fine-tuning can also weaken a model's safety behaviour, so safety tests must be rerun. Training data must be licensed and free of personal data unless there is consent, since models can memorise and repeat training examples.

How it works

  1. Try prompting with examples first and measure how far it gets.
  2. Add RAG if the problem is missing or changing facts.
  3. If style or skill is still lacking, collect high-quality input-output examples.
  4. Fine-tune with LoRA or QLoRA, keeping a held-out test set.
  5. Compare with the prompted base model, and rerun quality and safety tests.

The maths

LoRA update
Trainable parameters

For d = 4,096 and r = 8, one full matrix has 4,096² = 16,777,216 entries, while LoRA trains 2 × 4,096 × 8 = 65,536, which is 0.39 per cent. Applying LoRA to two matrices in each of 32 layers trains 32 × 2 × 65,536 = 4,194,304 parameters, about 0.06 per cent of a 7-billion-parameter model. In 16-bit precision the base model needs 7 billion × 2 bytes = 14 GB; in 4 bits it needs 3.5 GB. The LoRA adapter itself, at 16 bits, is about 4.2 million × 2 bytes ≈ 8 MB, small enough to email.

In code

lora_params.py
import numpy as np

# 1. Parameters trained by full fine-tuning versus LoRA for one 4096 x 4096 weight matrix
d, r = 4096, 8
full = d * d
lora = r * d + d * r                        # B is d x r, A is r x d
print(f"One matrix: full {full:,} vs LoRA rank {r}: {lora:,} ({lora / full:.2%})")
layers, adapted_per_layer = 32, 2           # e.g. the query and value projections in 32 layers
print(f"Whole 7-billion-parameter model: LoRA trains {layers * adapted_per_layer * lora:,} parameters, "
      f"{layers * adapted_per_layer * lora / 7e9:.3%} of the total")

# 2. Why low rank is enough: a weight change that is mostly low-rank plus small noise
rng = np.random.default_rng(0)
n = 256
true_update = rng.normal(size=(n, 4)) @ rng.normal(size=(4, n)) + 0.05 * rng.normal(size=(n, n))
U, S, Vt = np.linalg.svd(true_update)
for rank in [1, 2, 4, 8, 32]:
    approx = (U[:, :rank] * S[:rank]) @ Vt[:rank]
    err = np.linalg.norm(true_update - approx) / np.linalg.norm(true_update)
    print(f"rank {rank:>2}: {2 * n * rank:>6,} numbers instead of {n * n:,}; relative error {err:.3f}")

# 3. Memory for the weights alone, in gigabytes
for name, bytes_per in [("32-bit floats", 4), ("16-bit floats", 2), ("4-bit quantised (QLoRA)", 0.5)]:
    print(f"7B model in {name}: {7e9 * bytes_per / 1e9:.1f} GB")

Output

One matrix: full 16,777,216 vs LoRA rank 8: 65,536 (0.39%)
Whole 7-billion-parameter model: LoRA trains 4,194,304 parameters, 0.060% of the total
rank  1:    512 numbers instead of 65,536; relative error 0.837
rank  2:  1,024 numbers instead of 65,536; relative error 0.656
rank  4:  2,048 numbers instead of 65,536; relative error 0.025
rank  8:  4,096 numbers instead of 65,536; relative error 0.024
rank 32: 16,384 numbers instead of 65,536; relative error 0.020
7B model in 32-bit floats: 28.0 GB
7B model in 16-bit floats: 14.0 GB
7B model in 4-bit quantised (QLoRA): 3.5 GB

Try it yourself

Prompt, RAG or fine-tune?

Pick the best first approach for each need.

  • Answer staff questions from an HR policy updated monthly

  • Make replies sound friendly for a one-off campaign

  • Classify millions of support tickets cheaply

  • Write 50,000 product descriptions in a strict house style

  • Quote today's train timings

A legal assistant for Marathi court orders

A legal-tech startup wants summaries of Marathi court orders in a fixed English template.

Common mistakes

  • Fine-tuning to teach facts that change often instead of using retrieval.
  • Training on a few hundred careless examples and expecting good results.
  • Skipping safety and general-ability tests after fine-tuning.

Check yourself

Question 1

1. What does LoRA train?

Question 2

2. A matrix is 1,000 × 1,000 and r = 4. How many parameters does LoRA train for it?

Question 3

3. Your chatbot must answer from a price list that changes weekly. Best approach?

Summary

  • Adapt LLMs by prompting, retrieval (RAG) or fine-tuning; use fine-tuning for behaviour, not changing facts.
  • LoRA trains a small low-rank update and QLoRA adds 4-bit compression, making fine-tuning cheap.
  • Use high-quality examples, a held-out test set, and rerun safety checks.

Finished reading?

Sources: Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models · Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs · Hugging Face PEFT documentation (parameter-efficient fine-tuning)