Definition
Training a top image model from scratch can take millions of photos and weeks of GPU time. Yet a student with 200 photos of local plants can build an accurate classifier in an afternoon. The trick is to borrow what another model already learned.
Transfer learning reuses a model trained on one large task as the starting point for a related task. A pre-trained model, such as a CNN trained on ImageNet or a language model trained on web text, has learned general features. You can use it as a fixed feature extractor and train only a new output layer, or fine-tune some or all of its layers on your data with a small learning rate.
In simple words
Someone who already plays the guitar learns the ukulele much faster than a beginner, because finger skills and music sense carry over. A pre-trained model already knows edges, shapes and textures, or grammar and word meanings, so it only needs to learn the new specifics.
Overview
- Transfer learning reuses a model trained on one large task as the starting point for another.
- Early layers learn general features (edges, word meanings) that transfer well.
- It needs far less data and computing time than training from scratch.
- Models are shared through hubs such as Hugging Face, TensorFlow Hub and torchvision.
- The source and target tasks should be related for transfer to help.
- Licences and biases of pre-trained models carry over and must be checked.
Why it matters
Most real deep-learning projects use transfer learning. It cuts data needs from millions of examples to hundreds, training from weeks to minutes and cost from thousands of rupees to nearly nothing, which puts deep learning within reach of schools, small businesses and researchers.
Key terms
- Pre-trained model
- A model already trained on a large general data set.
- Feature extraction
- Freezing the pre-trained layers and training only a new head on top.
- Fine-tuning
- Continuing to train some or all pre-trained layers on new data with a small learning rate.
- Freezing
- Fixing layers' weights so training does not change them.
- Head
- The final task-specific layers added to a pre-trained backbone.
- Foundation model
- A very large pre-trained model adapted to many tasks, such as a large language model.
Types
Transfer learning comes in several types, from using a model as-is to retraining all of it.
1. Zero-shot use
Using a pre-trained model directly, without any further training.
- How it works: The model's general knowledge is applied through prompts or label descriptions.
- Example: CLIP labelling photos as "dog" or "bus" from text prompts; an LLM classifying reviews from instructions.
- When to use it: Use it first to see how far a strong model gets with no extra data.
2. Feature extraction
Freezing a pre-trained network and training only a new output layer on its features.
- How it works: The frozen base turns each input into a feature vector; a small classifier learns from these vectors.
- Example: Using ResNet features to classify 500 leaf photos into five diseases.
- When to use it: Use it when your data set is small and similar to the original training data.
3. Fine-tuning
Continuing to train some or all pre-trained layers on the new task, with a small learning rate.
- How it works: Usually the top layers are unfrozen first; the small rate avoids destroying the useful pre-trained features.
- Example: Fine-tuning BERT on 10,000 labelled Hindi news headlines for topic classification.
- When to use it: Use it when you have a moderate amount of data or the new task differs more from the original.
4. Parameter-efficient fine-tuning (adapters, LoRA)
Adding a small number of trainable parameters while keeping the base model frozen.
- How it works: Adapters insert small layers; LoRA learns low-rank weight updates, often under 1% of the parameters.
- Example: One base language model with separate LoRA adapters for legal, medical and customer-service styles.
- When to use it: Use it for large models where full fine-tuning is too costly.
5. Domain adaptation and knowledge distillation
Adapting a model to a shifted domain, or training a small student model to copy a large teacher.
- How it works: Domain adaptation continues pre-training on in-domain text or images; distillation trains the student on the teacher's soft predictions.
- Example: DistilBERT (2019) keeps about 97% of BERT's language understanding with 40% fewer parameters.
- When to use it: Use domain adaptation for specialised fields and distillation to deploy models on phones.
Explained in detail
Why features transfer
The early layers of an image network learn edges, colours and textures that are useful for almost any image. Middle layers learn shapes and parts. Only the last layers are specific to the original classes. So a network trained to recognise 1,000 ImageNet categories, such as dogs, cars and cups, already contains a rich visual vocabulary that helps with X-rays, crop leaves or handwritten Devanagari. Language models are similar: pre-training on billions of sentences teaches grammar, facts and word meanings that help with any text task.
Two strategies
| Strategy | What you train | When to use |
|---|---|---|
| Feature extraction | Only the new head; backbone frozen | Small data set, similar to the original data |
| Fine-tune top layers | The head plus the last few backbone layers | Moderate data, or a somewhat different domain |
| Full fine-tuning | All layers, with a small learning rate | Plenty of data, or a very different domain |
Fine-tuning uses a learning rate about ten times smaller than training from scratch, so the useful pre-trained weights are adjusted gently rather than overwritten. A common recipe is to train the head first with the backbone frozen, then unfreeze the top layers and continue at a lower rate. For large language models, parameter-efficient methods such as LoRA train small added matrices instead of all billions of weights, making fine-tuning possible on a single GPU.
Where to find pre-trained models
- Keras Applications and PyTorch torchvision include image models such as ResNet, MobileNet and EfficientNet with ImageNet weights.
- The Hugging Face Hub hosts hundreds of thousands of models for text, speech and vision, including Indian-language models such as IndicBERT.
- Check each model's licence, training data and model card before use: a model inherits the biases and gaps of its training data.
- Transfer works best when the domains are related; a model trained on everyday photos helps less with satellite radar images.
Transfer learning also works across tasks within the same domain. In the example on this page, a network trained to name digits is reused to decide whether a digit is odd, with only twenty labelled examples. Large language models take the idea furthest: one pre-trained model is adapted, through fine-tuning or just through prompts, to summarise, translate, classify and answer questions. Indian projects such as Bhashini and AI4Bharat fine-tune shared pre-trained models for many Indian languages instead of training a separate model from scratch for each one.
How it works
- Choose a pre-trained model suited to your data type and device.
- Replace its output layer with a new head for your classes.
- Freeze the backbone and train the head.
- Optionally unfreeze the top layers and fine-tune with a small learning rate.
- Evaluate on a held-out test set from your real setting.
In code
import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
X, y = load_digits(return_X_y=True)
X = X / 16.0
X_big, X_new, y_big, y_new = train_test_split(X, y, test_size=0.4, random_state=0)
# 1. Pre-train on a big task: recognise digits 0-9 from about 1,000 labelled images
pre = MLPClassifier(hidden_layer_sizes=(128, 32), max_iter=800, random_state=0).fit(X_big, y_big)
def frozen_backbone(data): # reuse the hidden layers, weights frozen
h = data
for W, b in zip(pre.coefs_[:-1], pre.intercepts_[:-1]):
h = np.maximum(0, h @ W + b)
return h
# 2. New, related task on new images: is the digit odd? Only 20 labelled examples.
is_odd = y_new % 2
scratch, transfer = [], []
for seed in range(5): # average over 5 random picks of 20 examples
X_s, X_te, y_s, y_te = train_test_split(X_new, is_odd, train_size=20, random_state=seed, stratify=is_odd)
scratch.append(LogisticRegression(max_iter=2000).fit(X_s, y_s).score(X_te, y_te))
head = LogisticRegression(max_iter=2000).fit(frozen_backbone(X_s), y_s) # train only a new head
transfer.append(head.score(frozen_backbone(X_te), y_te))
print(f"From scratch on raw pixels : {np.mean(scratch):.2f}")
print(f"New head on pre-trained layers: {np.mean(transfer):.2f}")Output
From scratch on raw pixels : 0.82 New head on pre-trained layers: 0.87
Try it yourself
A plant app in a weekend
A student club wants an app that recognises five medicinal plants from the school garden.
Which strategy?
Choose for each situation.
150 photos of fruit, similar to everyday photos
300 customer reviews to classify by mood
Satellite radar scans, very unlike normal photos, with 20,000 examples
50,000 labelled microscope images
Common mistakes
- Fine-tuning with a large learning rate and destroying the pre-trained features.
- Forgetting to preprocess inputs the same way as during pre-training.
- Ignoring the licence and biases of the pre-trained model.
Check yourself
Summary
- Transfer learning reuses features learned on a large task for a new, related task.
- Freeze the backbone and train a new head, then optionally fine-tune with a small learning rate.
- It cuts data, time and cost dramatically, but check licences and inherited biases.
Finished reading?
Sources: Hugging Face course: transfer learning and fine-tuning · Dive into Deep Learning (Zhang, Lipton, Li, Smola)