Transfer Learning

Transfer learning starts from a representation learned for another task. Reusing an architecture with random weights is not pretrained transfer, and loading default weights can trigger a network download.

Prerequisites

Complete Convolutional Neural Networks and Image Augmentation.

Learning goals

• distinguish feature extraction, fine-tuning, and training an architecture from random initialization.
• replace a classifier head without accidentally updating frozen parameters.
• pair a weight bundle with its declared preprocessing and dataset/license context.

Worked example

The executable path constructs ResNet-18 with weights=None, so it never downloads weights. It demonstrates the mechanics of freezing the backbone and replacing the head, but it does not claim transferred knowledge.

import torch
from torch import nn
from torchvision.models import resnet18

torch.manual_seed(44)
model = resnet18(weights=None)
for parameter in model.parameters():
    parameter.requires_grad = False
input_features = model.fc.in_features
model.fc = nn.Linear(input_features, 3)
images = torch.zeros((2, 3, 32, 32), dtype=torch.float32)
model.eval()
with torch.inference_mode():
    logits = model(images)

trainable = sum(parameter.numel() for parameter in model.parameters() if parameter.requires_grad)
print(f"input={tuple(images.shape)} logits={tuple(logits.shape)}")
print(f"head_in={input_features} trainable_parameters={trainable}")
print(f"downloaded_weights=False transferred_features=False")

Output:

input=(2, 3, 32, 32) logits=(2, 3)
head_in=512 trainable_parameters=1539
downloaded_weights=False transferred_features=False

A genuine pretrained path is illustrative and intentionally excluded from automated execution because DEFAULT may download weights into the local cache:

from torchvision.models import ResNet18_Weights, resnet18

weights = ResNet18_Weights.DEFAULT
model = resnet18(weights=weights)  # may download; review weight/data license terms
preprocess = weights.transforms()  # resize, crop, range, and normalization contract

For feature extraction, freeze the backbone and train the new head. For fine-tuning, unfreeze selected layers with an intentionally chosen learning rate after the head works. In both cases, fit task-specific choices on training/validation data and reserve test data for the fixed final pipeline.

Common mistakes

• Calling weights=None transfer learning even though no learned representation was transferred.
• Using pretrained weights but a different channel order or normalization recipe.
• Freezing parameters while leaving a frozen backbone's batch-normalization layers in unintended training mode.
• Assuming source-domain accuracy guarantees target-domain calibration, robustness, or fairness.
• Ignoring the weights' training-data provenance, permitted use, and supply-chain trust.

Practice

Explain when you would freeze the whole backbone, unfreeze the last block, or train from scratch. Name the evidence needed before deciding that the transferred model is better for a deployment domain.

Next steps

Continue with Vision Evaluation and Class Imbalance to evaluate the resulting classifier beyond overall accuracy.

Primary sources

Checked 2026-10-16: TorchVision models and pretrained weights, TorchVision ResNet-18 weights, PyTorch transfer-learning tutorial, and PyTorch module evaluation mode.