Skip to main content

Regularization and Dropout

Overfitting is an observed generalization gap: training performance improves while representative validation performance stalls or degrades. Regularization changes the learning problem to reduce that gap. It cannot repair a leaked split, a shifted target, or an unrepresentative validation set.

Prerequisites​

Complete Weight Initialization and understand the difference between training, validation, and test data.

Learning goals​

  • identify overfitting from paired training and validation curves.
  • explain dropout's different training and evaluation behavior.
  • distinguish weight decay, early stopping, data augmentation, and capacity control.

Worked example​

PyTorch dropout independently zeroes elements with probability (p) during training and scales survivors by (1/(1-p)). In evaluation mode it is the identity. With the fixed seed below, four of eight values survive and are doubled, preserving a mean of one in this particular small draw.

import torch
from torch import nn

torch.manual_seed(43)
dropout = nn.Dropout(p=0.5)
inputs = torch.ones(8, dtype=torch.float32)

dropout.train()
training_output = dropout(inputs)
dropout.eval()
evaluation_output = dropout(inputs)

print(f"train={training_output.tolist()}")
print(f"eval={evaluation_output.tolist()}")
print(f"train_mean={training_output.mean().item():.2f}")

Output:

train=[0.0, 2.0, 2.0, 0.0, 0.0, 0.0, 2.0, 2.0]
eval=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
train_mean=1.00

This train/eval mismatch is intentional. Forgetting model.eval() makes validation stochastic and changes its meaning. Conversely, leaving the model in evaluation mode while training silently disables dropout.

Other controls solve different problems:

  • smaller capacity limits functions the model can express.
  • weight decay biases toward smaller parameters; AdamW decouples this shrinkage from its adaptive gradient.
  • early stopping selects a checkpoint using validation evidence; the test set remains untouched.
  • data augmentation encodes label-preserving transformations and must match the domain.
  • more representative data can address variance and coverage, but not mislabeled targets or deployment mismatch automatically.

Tune regularization on training/validation data only. If many settings are tried against one validation set, selection can overfit that validation evidence too; nested or repeated evaluation may be needed for high-stakes comparisons.

Common mistakes​

  • Reporting only training loss and declaring regularization successful.
  • Applying dropout during validation or forgetting to re-enter training mode afterward.
  • Adding aggressive dropout everywhere without checking underfitting.
  • Using the test set for early stopping, architecture choice, or weight-decay selection.
  • Calling weight decay, L2 penalties, and all norm constraints interchangeable.

Practice​

Run the example with p=0.25 and several seeds. Explain why individual means vary for eight elements even though inverted dropout preserves the expectation. Sketch training and validation curves for overfitting and underfitting.

Next steps​

Continue with Batch and Layer Normalization to compare two state and axis contracts often confused with regularization.

Primary sources​

Checked 2026-10-16: PyTorch Dropout, Srivastava et al. on dropout, PyTorch AdamW, and PyTorch module evaluation mode.