Loss Functions and Objectives

A loss supplies the scalar signal differentiated by the optimizer. It is a training objective, not a complete definition of product value. The tensor contract—shape, dtype, domain, and reduction—must match the task.

Prerequisites

Complete Activation Functions. You should understand logits as unconstrained real-valued model outputs.

Learning goals

• match common prediction tasks to output and target shapes.
• explain why fused logit losses are numerically safer.
• separate per-example loss, batch reduction, regularization, and evaluation metrics.

Worked example

Binary cross-entropy for a logit $z$ and label $y \in \{0,1\}$ is computed stably by BCEWithLogitsLoss, which combines sigmoid and cross-entropy without first materializing probabilities near exactly 0 or 1.

import torch
from torch import nn

logits = torch.tensor([-2.0, 0.0, 2.0], dtype=torch.float32)
targets = torch.tensor([0.0, 1.0, 1.0], dtype=torch.float32)
losses = nn.BCEWithLogitsLoss(reduction="none")(logits, targets)

print(f"losses={[round(value, 6) for value in losses.tolist()]}")
print(f"mean={losses.mean().item():.6f}")

Output:

losses=[0.126928, 0.693147, 0.126928]
mean=0.315668

Common contracts are:

Task	Model output	Target	Typical loss
binary classification	logits (batch,) or (batch, 1)	floats with identical shape	BCEWithLogitsLoss
single-label, (C)-class	logits (batch, C)	class indices (batch,), torch.long	CrossEntropyLoss
multi-label, (C)-class	logits (batch, C)	independent binary floats (batch, C)	BCEWithLogitsLoss
regression	predictions matching target shape	floating point	task-appropriate L1/MSE/other loss

CrossEntropyLoss expects unnormalized logits and accepts either class indices or valid class-probability targets. The two target modes have different shape and dtype contracts. PyTorch does not validate that probability targets form distributions, so callers retain that responsibility.

Reduction matters. A mean makes gradient scale less dependent on batch size; a sum changes it with the number of examples. Class weights, example weights, label smoothing, and regularization change the optimized estimand and should be recorded. Evaluate with metrics aligned to the decision—such as log loss, calibration, recall at a declared threshold, or task-specific cost—rather than assuming the training loss answers every question.

Common mistakes

• Applying sigmoid before BCEWithLogitsLoss or softmax before CrossEntropyLoss.
• Passing one-hot floats where class-index targets were intended, or squeezing away the batch dimension for batch size one.
• Comparing summed training loss with mean validation loss.
• Selecting a threshold on the test set.
• Treating lower surrogate loss as guaranteed improvement in every deployment outcome or subgroup.

Practice

For logits with shape (32, 5), state the target shape and dtype for single-label classification and for multi-label classification. Then explain which axis softmax would normalize and why that transformation belongs outside the CrossEntropyLoss input.

Next steps

Continue with Gradient Descent and Optimizers to use loss gradients to update parameters.

Primary sources

Checked 2026-10-16: PyTorch BCEWithLogitsLoss, PyTorch CrossEntropyLoss, and PyTorch loss functions.