Transfer Learning and Self-Supervised Learning
Reuse pretrained networks: freeze or fine-tune, catastrophic forgetting, and learning without labels through masking, SimCLR, CLIP and linear probes.
Intermediate lesson, about 40 minutes, with interactive demos and a quiz.
What you will learn
- Pretrained features
- Feature extraction vs fine-tuning
- Catastrophic forgetting
- Masked prediction
- Contrastive learning
- Linear probes
Reuse what a network already knows
A start-up wants to spot cracked solar panels in drone photos. A charity wants to recognise diseased cassava leaves. A hospital wants to flag fractures. None of them has a million labelled images, and none needs one. They start from a network that has already learned to see, and teach it only what is new.
What is it?
Transfer learning reuses a model trained on one task, the source, as the starting point for another, the target. The trained layers are the backbone; the small final layer that turns features into the target’s answers is the head. In vision the classic source is ImageNet classification: 1.28 million labelled photos in 1,000 classes.
How does it work?
A network trained on lots of varied data has to learn features that are useful far beyond its own labels. The Convolutional Neural Networks lesson shows the hierarchy that emerges: oriented edges and colour blobs first, then textures, then parts, then whole objects. Yosinski and colleagues measured how far each layer carries to a new task. They found that the features “must eventually transition from general to specific by the last layer”: early layers transfer almost anywhere, later ones less so, and the gap grows as the source and target tasks drift apart.
Why does it matter?
- Labels are the expensive part. In 2014, Razavian and colleagues took 4,096 features from an ImageNet network, added only a linear classifier, and beat carefully tuned specialist systems across a diverse set of recognition tasks. Oquab and colleagues showed the same layers transfer to new datasets such as Pascal VOC. Since then, most teams never train a vision model from scratch.
- It is how foundation models are used. Large language models, CLIP and modern vision backbones are pretrained once at enormous cost and then adapted many times, cheaply.
- It is not magic. When the target domain is very different and in-domain labels are plentiful, the benefit can shrink to little or nothing, as the next section shows.
Key takeaways
- Transfer learning reuses a backbone trained on a large source task and trains a new head for the target task.
- Early layers learn general features such as edges and textures; later layers become specific to the source task.
- Pretrained features let small labelled datasets go much further, which is why most vision projects start from one.
Freeze or fine-tune?
There are two ways to use a pretrained backbone. Which wins depends mostly on two things: how many labels you have, and how far your task is from the source.
What is it?
Feature extraction freezes the backbone and trains only a new head on its outputs. Full fine-tuning starts from the pretrained weights and keeps training every layer, so the features themselves can change. Both are compared with training the same network from scratch.
How does it work?
Few labels, similar domainFreeze and train a head. There is too little data to improve the features safely, and they already fit.Many labels, similar domainFine-tune. There is enough data to adjust the features to the details of your task.Few labels, distant domainHardest case. Keep early layers, fine-tune the rest gently, use strong augmentation, or find a closer pretrained model.Many labels, distant domainFine-tune, and check against training from scratch: the gain from pretraining may be small.When fine-tuning, use a learning rate well below the one used in pretraining. The new head starts random, so its first gradients are large and noisy, and a big step can wreck good features before the head has learned anything. Common refinements are smaller rates for earlier layers and unfreezing layers one at a time from the top, as in ULMFiT, or training the head first with the backbone frozen and only then fine-tuning everything. Kumar and colleagues called the last recipe LP-FT and found it beat both plain fine-tuning and a frozen backbone, especially on data unlike the training set.
On the related task, the frozen backbone leads while labels are scarce (73% with 4 labels, against 62% for fine-tuning and 56% from scratch) but stalls in the mid 80s, because its features were built for a different flower. Fine-tuning and training from scratch keep improving, and by 128 labels they have passed it, with fine-tuning at 92% and from scratch at 93%. That is the general pattern: pretraining matters most when labels are few, and less as they grow. On the unrelated stripes the frozen features are close to useless, while fine-tuning recovers.
The demo also shows the price of fine-tuning. The flower accuracy of the frozen model never changes (98%), but after fine-tuning on 4 or 8 labels it falls to 74 to 80% with the small learning rate, and to 55 to 69% with the large one. The network has overwritten what it knew.
Why does it matter?
These choices decide whether a small project works. Raghu and colleagues found that on two large medical imaging tasks, ImageNet pretraining added little accuracy, and small models trained from scratch did about as well. With a few hundred images, by contrast, a pretrained backbone is often the difference between a usable model and a useless one. Always compare against a frozen baseline on held-out data.
Key takeaways
- Freezing the backbone and training a head is safest with few labels and a similar domain; fine-tuning wins with more labels or a bigger domain gap.
- Fine-tune with a much smaller learning rate than pretraining, and consider training the head first.
- Fine-tuning can erase the old task (catastrophic forgetting); freeze, add adapters or rehearse old data if you need both.
Labels from the data itself
ImageNet took years of human labelling. The web holds billions of unlabelled images and trillions of words. Self-supervised learning finds a way to pretrain on all of it.
What is it?
Self-supervised learning invents a training task whose answers are already in the data, so no person has to label anything. The task itself is not the goal. It is a pretext task, chosen because solving it forces the network to learn features that are useful for the real tasks later.
How does it work?
Early pretext tasks for images were puzzles:
- Rotation. Turn a photo by 0°, 90°, 180° or 270° and ask which. To answer, the network must know that skies are above and feet are below (Gidaris et al., 2018).
- Jigsaw. Cut a photo into nine tiles, shuffle them, and ask for the original order, which needs a sense of object parts (Noroozi and Favaro, 2016).
The idea that took over was simpler: hide part of the input and predict it. BERT hides 15% of the words in a sentence and predicts them from the words on both sides. Masked autoencoders (MAE) cut an image into patches, hide a much larger share, typically 75%, encode only the visible patches, and reconstruct the pixels of the hidden ones. Next-token prediction, the objective behind every large language model, is the same idea applied to the end of a text.
Why does it matter?
Self-supervision removed the label bottleneck from pretraining. It is how language models are pretrained, and how many of the strongest vision backbones are built: the Computer Vision lesson describes DINOv3, trained without labels on about 1.7 billion images. Labels are still needed, but only a few, at the end, for the target task.
Large Language Models: next-token prediction as self-supervised pretraining at the scale of the web.Key takeaways
- Self-supervised learning creates training targets from the data itself, so pretraining needs no human labels.
- Pretext tasks such as rotation or jigsaw puzzles force the network to learn useful structure.
- Masked prediction (BERT: 15% of words; MAE: 75% of image patches) became the dominant recipe.
Contrastive learning: SimCLR
Another way to learn without labels: take two random edits of the same photo and teach the network that they show the same thing, and that every other photo in the batch shows something different.
What is it?
Contrastive learning trains an encoder so that matching pairs, called positives, land close together in embedding space and non-matching pairs, negatives, land far apart. In SimCLR, Chen and colleagues made each positive pair from one image by applying two random augmentations (crops, flips, colour distortion, blur). Every other image in the batch supplies the negatives.
How does it work?
Each augmented view goes through the encoder and a small projection head, and the output is scaled to length 1, so the similarity of two views is the cosine of the angle between them. For each view, the loss is a softmax over the similarities to every other view in the batch, with the positive as the right answer. This is the InfoNCE loss, named by van den Oord and colleagues; SimCLR’s version is called NT-Xent.
ℓ(i) = −log [ exp(sim(zᵢ, zⱼ)/τ) / Σk≠i exp(sim(zᵢ, zₖ)/τ) ]For a view i with positive j, among 2N views in a batch of N images. sim is cosine similarity and τ is the temperature.
The intuition is a multiple-choice test: out of all these views, which one came from the same image as yours? The only way to pass reliably is to keep what survives the augmentations, the content, and discard what they change. The temperature τ sets how sharply the softmax focuses on the hardest negatives. Augmentations decide what the network learns to ignore: SimCLR found the composition of augmentations, particularly random cropping with colour distortion, mattered most.
With rotation up to ±180°, the loss first sits near log 127 ≈ 4.84, the value for random guessing, while every embedding points almost the same way. Then it drops, the three rings separate into three arcs, and a linear probe reads the ring from the embedding nearly perfectly, although the raw input scores about 30%. Nothing told the encoder about rings. It learned radius because radius is what two rotated views share. At τ = 1 the softmax is too soft to push negatives apart firmly, and the probe usually ends lower and less steady.
Why does it matter?
Contrastive learning turned unlabelled images into strong general features, and it is the engine inside CLIP and many embedding models. Its lesson generalises: a self-supervised model learns exactly the invariances its training task rewards, so the choice of augmentations, or of what to mask, is a modelling decision.
Key takeaways
- Contrastive learning pulls positives (two augmented views of one example) together and pushes negatives (everything else in the batch) apart.
- InfoNCE is a softmax that must pick the positive out of the batch; the temperature sets how sharply it weighs hard negatives.
- Augmentations decide what the representation ignores, so they must not change what you later want to predict.
CLIP and zero-shot classification
Swap “two views of one image” for “an image and its caption”, and contrastive learning connects vision to language.
What is it?
CLIP trains an image encoder and a text encoder together on 400 million image-caption pairs from the web, with the same kind of InfoNCE loss: each image must pick out its own caption from the batch, and each caption its own image. The positives come from the data, so this is self-supervised in spirit, with the web’s captions as a free, noisy label.
How does it work?
Because images and text share one embedding space, you can classify an image without training a classifier. Write a caption for each class (“a photo of a dog”), embed them, and pick the caption closest to the image. This is zero-shot classification: no labelled examples of the target task at all. Zero-shot CLIP matched the ImageNet accuracy of a supervised ResNet-50 without using any of its 1.28 million training labels.
Why does it matter?
Zero-shotNo labels; describe the classes in words. Works when the distinction is something web captions talk about.Linear probeA few labels; train a linear head on frozen CLIP features. Often a large step up on specialist tasks.Fine-tuneMore labels; adapt the encoder itself. Best accuracy, with the forgetting and overfitting risks above.The three form a ladder from no labels to many. The Multimodal AI lesson trains a tiny CLIP and lets you run zero-shot classification yourself, including how adding a label changes the answer.
Embeddings and Retrieval: once images or text are vectors, search is a nearest-neighbour lookup.Key takeaways
- CLIP applies contrastive learning across images and their captions, putting both in one embedding space.
- Zero-shot classification compares an image with text descriptions of the classes, with no task-specific labels.
- With a few labels, a linear probe on CLIP features usually beats zero-shot; with more, fine-tuning can go further.
Linear probes: testing a representation
Every self-supervised method claims to learn good features. How do you check, when the pretraining task itself is not what you care about?
What is it?
A linear probe freezes the pretrained network, takes its features for a labelled dataset, and trains only a linear classifier on top. Its held-out accuracy measures how much of the label is readable from the features with a straight-line boundary. Alain and Bengio used probes to examine each layer of a network; linear evaluation on ImageNet became the standard scoreboard for self-supervised vision, used by SimCLR and CLIP among many others.
How does it work?
The classifier is deliberately weak. If a straight line can separate the classes in feature space, the network has done the hard work of untangling them. You met one already: in the SimCLR demo, the same linear probe scores about 30% on the raw points and close to 100% on the learned embedding. The probe did not get smarter; the features did. Feature extraction, from the previous sections, is a linear probe used in production.
What a probe does not tell you
A probe measures linear readability, not everything. MAE’s ViT-L scored 75.8% under a linear probe, below the 77.6% of a contrastive method (MoCo v3), yet MAE fine-tuned better. Its authors found linear probing and fine-tuning results were largely uncorrelated, and argued that probes miss “strong but non-linear features”. Report probes alongside fine-tuning and few-label results, and evaluate on the kind of data you will actually use.
Why does it matter?
Probes are cheap, fast and hard to game, which makes them the common yardstick for comparing pretrained models. When you choose a backbone for your own project, a linear probe on a few hundred of your labelled images is often the quickest honest test.
Interpretability uses the same probes to ask what a network represents inside.Key takeaways
- A linear probe trains only a linear classifier on frozen features; its held-out accuracy scores the representation.
- High probe accuracy means the network has already untangled the classes; the probe itself is deliberately simple.
- Probe and fine-tuning rankings can disagree, so use both, on data like yours.
Check your understanding
Seven situations where reusing a pretrained model helps, hurts or needs care.
Question 1 of 7A plant charity has 300 labelled photos of leaves, half healthy and half with a disease. They have an ImageNet-pretrained network and a laptop. What is the best first thing to try?
Next, see a pretrained network at work in Object Detection, where detectors are built on backbones pretrained for classification.
References
Papers behind the methods and claims in this lesson.
References
- [1]
How transferable are features in deep neural networks?(opens in a new tab)
Jason Yosinski, Jeff Clune, Yoshua Bengio & Hod Lipson, 2014
NeurIPS 2014. Measures, layer by layer, how general or task-specific an ImageNet network’s features are, and finds that transferred features help even after fine-tuning.
- [2]
CNN features off-the-shelf: an astounding baseline for recognition(opens in a new tab)
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan & Stefan Carlsson, 2014
CVPR Workshops 2014. 4,096 features from an ImageNet-trained network (OverFeat) plus a linear SVM beat tuned specialist systems on a diverse set of recognition tasks.
- [3]
Maxime Oquab, Léon Bottou, Ivan Laptev & Josef Sivic, 2014
CVPR 2014. Reuses the layers of an ImageNet network, adds new adaptation layers, and transfers to Pascal VOC object and action classification.
- [4]
Universal language model fine-tuning for text classification(opens in a new tab)
Jeremy Howard & Sebastian Ruder, 2018
ACL 2018. ULMFiT: discriminative learning rates (smaller for earlier layers) and gradual unfreezing when fine-tuning a pretrained network.
- [5]
Fine-tuning can distort pretrained features and underperform out-of-distribution(opens in a new tab)
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma & Percy Liang, 2022
ICLR 2022. Across 10 distribution-shift datasets, full fine-tuning beat linear probing by about 2% in distribution but lost by about 7% out of distribution; training the head first (LP-FT) did better than both.
- [6]
Michael McCloskey & Neal J. Cohen, 1989
Psychology of Learning and Motivation 24. Shows that training a network on new material can abruptly erase what it learned before.
- [7]
Transfusion: understanding transfer learning for medical imaging(opens in a new tab)
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg & Samy Bengio, 2019
NeurIPS 2019. On two large medical imaging tasks, ImageNet pretraining gave little accuracy benefit; small models trained from scratch did about as well.
- [8]
Unsupervised representation learning by predicting image rotations(opens in a new tab)
Spyros Gidaris, Praveer Singh & Nikos Komodakis, 2018
ICLR 2018. A pretext task: rotate an image by 0, 90, 180 or 270 degrees and train the network to say which.
- [9]
Unsupervised learning of visual representations by solving jigsaw puzzles(opens in a new tab)
Mehdi Noroozi & Paolo Favaro, 2016
ECCV 2016. A pretext task: cut an image into a 3 × 3 grid of tiles, shuffle them, and train the network to recover the order.
- [10]
BERT: pre-training of deep bidirectional transformers for language understanding(opens in a new tab)
Jacob Devlin, Ming-Wei Chang, Kenton Lee & Kristina Toutanova, 2019
NAACL 2019. Masked language modelling: hide 15% of the tokens and train a transformer to predict them from both sides.
- [11]
Masked autoencoders are scalable vision learners(opens in a new tab)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár & Ross Girshick, 2022
CVPR 2022. MAE: hide 75% of an image’s patches, encode only the visible ones, and reconstruct the pixels of the rest. Also compares linear probing with fine-tuning.
- [12]
A simple framework for contrastive learning of visual representations(opens in a new tab)
Ting Chen, Simon Kornblith, Mohammad Norouzi & Geoffrey Hinton, 2020
ICML 2020. SimCLR: two random augmentations per image, a projection head and the NT-Xent loss. A linear classifier on its features reached 76.5% top-1 on ImageNet.
- [13]
Representation learning with contrastive predictive coding(opens in a new tab)
Aaron van den Oord, Yazhe Li & Oriol Vinyals, 2018
arXiv. Introduces the InfoNCE loss: pick the true partner out of a set of negatives with a softmax.
- [14]
Learning transferable visual models from natural language supervision(opens in a new tab)
Alec Radford, Jong Wook Kim, Chris Hallacy et al., 2021
ICML 2021. CLIP: contrastive training on 400 million image-text pairs, zero-shot classification from text prompts, and linear-probe comparisons across many datasets.
- [15]
Understanding intermediate layers using linear classifier probes(opens in a new tab)
Guillaume Alain & Yoshua Bengio, 2016
arXiv. Trains a linear classifier on each layer’s activations to measure how linearly readable the information there is.
Related
- Builds on: Convolutional Neural Networks
- Builds on: Neural Networks and Deep Learning
- Practise in the lab: Object Detection Lab
- Practise in the lab: Embedding Explorer