Skip to main content

Pretraining and Transfer Learning

Pretraining learns parameters from a broad source corpus using a self-supervised objective such as masked-token or next-token prediction. Transfer learning initializes a downstream task from those parameters. The downstream result still depends on source data, tokenizer, objective, model version, and domain match.

Prerequisites​

Complete Transformer Architecture and understand logits and cross-entropy.

Learning goals​

  • distinguish masked-token, causal next-token, and downstream supervised objectives.
  • encode supervision only at selected positions with an ignored-label sentinel.
  • make pretrained artifacts, licenses, model cards, and downloads explicit dependencies.

Worked example​

This local tensor illustrates masked-language-model loss. -100 is PyTorch's default ignored target for CrossEntropyLoss; only the two masked positions contribute.

import torch
from torch import nn

vocabulary_size = 6
input_ids = torch.tensor([[2, 1, 4, 0], [3, 5, 1, 0]])  # ID 1 is <mask>, ID 0 is <pad>
labels = torch.tensor([[-100, 3, -100, -100], [-100, -100, 2, -100]])
logits = torch.zeros(2, 4, vocabulary_size)
logits[0, 1, 3] = 2.0
logits[1, 2, 2] = 2.0

loss = nn.CrossEntropyLoss(ignore_index=-100)(logits.reshape(-1, vocabulary_size), labels.reshape(-1))
masked = labels.ne(-100)
predictions = logits.argmax(dim=-1)
accuracy = predictions[masked].eq(labels[masked]).float().mean()

print(f"input={tuple(input_ids.shape)} logits={tuple(logits.shape)}")
print(f"masked_positions={masked.nonzero().tolist()} supervised_tokens={masked.sum().item()}")
print(f"loss={loss.item():.4f} masked_accuracy={accuracy.item():.1f}")

Output:

input=(2, 4) logits=(2, 4, 6)
masked_positions=[[0, 1], [1, 2]] supervised_tokens=2
loss=0.5168 masked_accuracy=1.0

The logits are constructed, not learned; the example verifies objective mechanics only. A masked encoder can use context on both sides. A causal model predicts each next token without seeing future tokens. Neither objective guarantees factual, unbiased, or task-relevant representations.

The following Hugging Face boundary is manual only. It is excluded from CI because from_pretrained may require cached or downloaded tokenizer and weight artifacts. Use transformers==5.20.0 in a separate environment, review the model card and license, pin an immutable revision, and record artifact hashes.

from transformers import AutoModel, AutoTokenizer

model_id = "distilbert/distilbert-base-uncased"
revision = "PIN_A_REVIEWED_COMMIT_SHA"
tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)
model = AutoModel.from_pretrained(model_id, revision=revision)

This is a manual/download boundary, not evidence that the artifact is already available or that its output was verified here.

Common mistakes​

  • Calling self-supervised pretraining unsupervised without explaining how targets are constructed.
  • Comparing pretraining losses across different tokenizers as if they had the same units.
  • Assuming a popular checkpoint is licensed, safe, current, or suitable for the deployment domain.
  • Evaluating repeatedly on the test split while selecting a checkpoint or model family.
  • Loading mutable remote model IDs without recording revisions and hashes.

Practice​

Change one masked target to -100 and predict the number of supervised tokens and loss behavior. Then write an artifact manifest for a pretrained extension: model ID, revision, tokenizer files, license, source URL, and checksum.

Next steps​

Fine-Tuning a Text Classifier adapts representations to labeled examples.

Primary sources​

Checked 2026-10-16: BERT paper, GPT paper, PyTorch CrossEntropyLoss, Hugging Face Auto Classes, and Hugging Face model-sharing guidance.